Crowdsourcing has rapidly become a computing paradigm in machine learning and artificial intelligence. In crowdsourcing, multiple labels are collected from crowd-workers on an instance usually through the Internet. These labels are then aggregated as a single label to match the ground truth of the instance. Due to its open nature, human workers in crowdsourcing usually come with various levels of knowledge and socio-economic backgrounds. Effectively handling such human factors has been a focus in the study and applications of crowdsourcing. For example, Bi et al studied the impacts of worker's dedication, expertise, judgment, and task difficulty (Bi et al 2014). Qiu et al offered methods for selecting workers based on behavior prediction (Qiu et al 2016). Barbosa and Chen suggested rehumanizing crowdsourcing to deal with human biases (Barbosa 2019). Checco et al studied adversarial attacks on crowdsourcing for quality control (Checco et al 2020). There are many more related works available in literature. In contrast to commonly used binary-valued labels, interval-valued labels (IVLs) have been introduced very recently (Hu et al 2021). Applying statistical and probabilistic properties of interval-valued datasets, Spurling et al quantitatively defined worker's reliability in four measures: correctness, confidence, stability, and predictability (Spurling et al 2021). Calculating these measures, except correctness, does not require the ground truth of each instance but only worker’s IVLs. Applying these quantified reliability measures, people have significantly improved the overall quality of crowdsourcing (Spurling et al 2022). However, in real world applications, the reliability of a worker may vary from time to time rather than a constant. It is necessary to monitor worker’s reliability dynamically. Because a worker j labels instances sequentially, we treat j’s IVLs as an interval-valued time series in our approach. Assuming j’s reliability relies on the IVLs within a time window only, we calculate j’s reliability measures with the IVLs in the current time window. Moving the time window forward with our proposed practical strategies, we can monitor j’s reliability dynamically. Furthermore, the four reliability measures derived from IVLs are time varying too. With regression analysis, we can separate each reliability measure as an explainable trend and possible errors. To validate our approaches, we use four real world benchmark datasets in our computational experiments. Here are the main findings. The reliability weighted interval majority voting (WIMV) and weighted preferred matching probability (WPMP) schemes consistently overperform the base schemes in terms of much higher accuracy, precision, recall, and F1-score. Note: the base schemes are majority voting (MV), interval majority voting (IMV), and preferred matching probability (PMP). Through monitoring worker’s reliability, our computational experiments have successfully identified possible attackers. By removing identified attackers, we have ensured the quality. We have also examined the impact of window size selection. It is necessary to monitor worker’s reliability dynamically, and our computational results evident the potential success of our approaches.This work is partially supported by the US National Science Foundation through the grant award NSF/OIA-1946391.
Crowdsourcing has become a popular paradigm in machine learning. It gath- ers labels from large groups of people (crowd-workers), usually through the internet.
The quality of crowdsourcing very much depends on the quality of the collected labels in addition to learning algorithms (Sheng and Zhang, 2019). Due to its open nature, human factors are unavoidably involved in crowd- sourcing. Barbosa and Chen reported that human biases caused by worker’s social-economic status can determine the outcome of crowdsourcing tasks (Barbosa and Chen, 2019). Another issue is that the level of expertise for crowd-workers often varies widely. Workers with a high level of expertise usually give better quality labels than those with less. However, a highly knowledgeable worker with adversarial intentions can cause more harm. Pro- perly managing human factors in crowdsourcing has been actively studied. For instance, Bi et al studied the impacts of worker’s dedication, expertise, judgment, and task difficulty in (Bi et al. 2014). Qiu et al offered methods for selecting workers based on behavior prediction (Qiu et al. 2016). Tao et al reported quality improvements in (Tao et al. 2020) when utilizing MV-Freq and MV-Beta (Sheng et al. 2019) with worker’s reliability.
1This work is partially supported by the US National Science Foundation through the grant award NSF/OIA-1946391.
Dynamically Monitoring Crowd-Worker’s Reliability with Interval-Valued Labels 23
In contrast to commonly used binary-valued labels in previous work, Hu et al proposed interval-valued labels (IVLs) recently (Hu et al. 2021). Appl- ying statistical and probabilistic properties of interval-valued datasets, one can quantify worker’s reliability and achieve significant quality improvement (Spurling et al., 2021) and (Spurling et al., 2022). Worker’s behavior in real world often varies from time to time and is not consistent as implicitly assu- med in previous study. We should monitor and update workers’ reliability dynamically.
A BRIEF REVIEW OF PREVIOUS WORK Prior to our discussion, let us briefly review related concepts, notations, and previous results as background knowledge first.
Statistic and Probabilistic Properties of an Interval-Valued Dataset
a + a
a −a
In this paper, we use interval-valued labels. Following the literature of interval computing, we use boldface letters to separate interval-valued objects from point-valued ones. For example, a is a real where a is an interval. The greatest lower and least upper bounds of an interval are specified with an underline and an overline on the same non-boldface letter. Thus, a = [a, a] . The mid- point and radius of a are mid (a) =
Equations and mathematical formulas are available in the original paper PDF.
/2. As the two are point-valued, we write them without boldface as mid (a) and rad (a) hereafter.
Let L= [l1, l2, · · · , lm] be a list of intervals. The midpoint and radius of L are mid (L) and rad (L), respectively. The mean of L is the interval: #
"Pm
Pm
m X
m ,
m
m X
mid(L)
+ Var
rad(L)
−µ
−µ
rad (L)
mid (L)
li
li
. So, we can calculate the standard deviation of L as usual:
Hu and Hu (2020a) also defined a probability density function (pdf) for L as
Pm
where pdfi (t) is the pdf of an li ∈L. With the pdf, we can calculate Shannon’s entropy of L.
24 Hu and Spurling
Reliability Indicators Derived from Crowd-Worker’s IVLs We assume a binary classification model in this work. Let V be a set of obse- rvations. The objective of a crowdsourced task is to determine whether a vi ∈V is an instance of a given class y or not with multi-labels provided by some j ∈J, which is the set of crowd-workers. We use lj
i to denote the IVL by a worker j ∈J for a given vi ∈V. Because of the binary classification i h model, lj ⊆[0, 1]. The IVL contains j’s uncertainty on vi. A real is j i i = lj −i, l
in fact a narrow interval with its greatest lower and least upper bounds the same. IVLs extend binary-valued labels 0 and 1.
g . Then, the
Let Lj be the list of IVLs made by a worker j ∈J. From which, one may quantify j’s reliability into four reliability indicators: correctness, confidence, stability, and predictability (Spurling et al., 2021). The correctness of a wor- ker is the ratio of accurately labeled observations by the worker. A set of gold questions with known ground truth is commonly used to estimate a worker’s correctness. Let G = g be a set of gold questions. The IVL from a wor- ker j on a g ∈G is lj
g. Denoting the ground truth of g as o
center-correctness of lj
Equations and mathematical formulas are available in the original paper PDF.
mid
lj
Without loss of generality, we can assume the ground truth is 1 for any g ∈G in calculating j’s correctness. This is because of that, when o
g
we can replace lj
g
1 −l j g, 1 −j
g with its difference from 1, i.e. 1 −lj without changing the center-correctness. Let Lj
G be j’s IVLs on the set of gold questions G. We can apply Eq. (1) to calculate the mean of Lj G, µ(Lj G), which estimates j’s average correctness.
lj
An IVL lj
i
i contains information of j’s confidence too. When mid j has absolutely no confidence toward either 0 or 1. Otherwise, mid lj reprei
lj
−0.5 i
i. Additionally, the radius
sents j’s preference toward 0 or 1. The distance between the midpoint and 0.5, i.e., mid
, reflects confidence of lj
of lj
i specifies the maximum possible variation from the midpoint. When rad lj = 0, the label is point-valued. Otherwise, the label contains j’s i
uncertainty over a range. The maximum possible value of rad
lj
is 0.5. The i
difference between 0.5 and rad
lj
, 0.5−rad
lj
, is another measure of coni i
fidence. Ultimately, the confidence of a single lj
i is a combination of the above two. That is:
lj
lj
lj
i
i i
mid
+ 0.5 −rad
Dynamically Monitoring Crowd-Worker’s Reliability with Interval-Valued Labels 25
Figure 1: The relation between worker’s correctness and confidence (Spurling et al. 2021).
Since both
lj
lj
are between 0 and 0.5, the
i
i
mid
and 0.5 −rad
confidence of lj
i can be any real between 0 and 1. It is important to notice that the confidence of an IVL does not depend on the ground truth. Thus, the mean of Lj reflects j’s overall confidence as
µ
µ
Equations and mathematical formulas are available in the original paper PDF.
mid
+ 0.5 −rad
Fig. 1 illustrates the relationship between worker’s correctness and confi- dence (Spurling et al. 2021), i.e., one’s correctness and confidence should be a point inside the V shaped area.
The overall stability of j is reflected in σ
Lj
, which can be calculated with Eqs. (2) and (3). In addition, Eq. (4) provides a pdf of Lj. Hence, we can calculate Shannon’s entropy of Lj as an estimation of j’s predictability. It is important to notice that estimating j’s confidence, calculating j’s stability and predictability does not require the ground truth of a v ∈V.
DYNAMICALLY MONITORING WORKER’S RELIABILITY In this section, we discuss the needs and approaches to monitor worker’s reliability dynamically.
We Need to Monitor Worker’s Reliability Dynamically Applying the quantified reliability measures above, we can effectively take human factors into consideration. For instance, reliability weighted infere- nce making schemes such as weighted interval majority voting (WIVM) and weighted preferred matching probability (WPMP) can significantly improve
26 Hu and Spurling
overall quality (Spurling et al., 2021). A challenging task in crowdsourcing is to identify and exclude labels from those who are not reliable even with adversary purposes. In fact, attackers may pretend to be regular workers with the sole purpose to derail a crowdsourced task (Checco et al., 2020, Wang et al. 2014). Very sophisticated attackers are likely to identify gold questions and answer them with a high level of correctness, and deliberately label regu- lar questions incorrectly (Qiu et al. 2016). Monitoring worker’s reliability derived from IVLs, one may identify such anomalous workers.
Let G be the set of gold questions and U be the set of regular questions. We can partition Lj as Lj
U. Assuming G well samples U. Then, Lj
G and Lj
G and Lj
U should be consistent statistically when j behaves normally. Otherwise, it implies possible anomaly. Applying the t-test and F-test, we can examine the consistency between Lj
G and Lj
U statistically without the ground truth of any u ∈U. With this approach, Spurling et al. (2022) successfully iden- tified attackers and achieved significant quality improvements. However, it was implicitly assumed that the reliability of a given worker is constant. This is not generally true in real world applications. For instance, a worker with malicious intention may purposely change his/her labelling pattern from time to time. A dedicated worker may improve his/her reliability along with more experience/knowledge. Moreover, random factors such as changing working environment, emotion, stress level, and others can cause variations of wor- ker’s reliability. Therefore, we need to monitor and update worker’s reliability dynamically.
Ways of Monitoring and Updating Worker’s Reliability Dynamically
i
h
is
lj
The reliability of a worker j varies from time to time. Thus, j’s reliability is a function of time rj (t). The reliability of j is derived from the list of his/her IVLs Lj. Because j labels vi ∈V sequentially, Lj =
k + 1, · · ·
1,lj 2, · · · ,lj k,lj
in terms of correctness, confidence, stability, and
lj
an interval-valued time series. To monitor worker’s behavior dynamically, we assume that the reliability value of j, rj, at time t depends on some con- secutive IVLs in Lj within a time window T. Assuming the first k IVLs in Lj are in the window T0. We can then initiate j’s reliability at t = t0 as rj (t0) = f
1,lj 2, · · · ,lj k predictability. As discussed earlier, the correctness of j’s reliability relies on j’s IVLs on gold questions. To monitor j’s correctness dynamically, we need to present gold questions periodically. When the IVL on a new gold question from j becomes available, we can re-evaluate j’s correctness and other reliabi- lity indicators. Let lj
g, be the IVL for a new g ∈G at t0 < t ≤t1 by a worker j. To update j’s reliability rj (t1), we use IVLs in the time window T1 only, which include the newly available lj
g. Continuing the process, we can dynamically evaluate j’s current reliability in a moving time window Ti. (Note: worker’s inactive time should be excluded.) One should apply worker’s current reli- ability in inference making. For anomaly detection, one needs to examine and compare the reliability derived from the IVLs on gold questions Lj
GTiand
from those on regular questions Lj
UTiwithin the time window Ti.
Dynamically Monitoring Crowd-Worker’s Reliability with Interval-Valued Labels 27
As the time window moves forward, the worker j’s correctness, confidence, stability, and predictability are updated. Each of these four reliability indi- cators forms a time series too. With a user pre-selected orthogonal basis of function space, say f0, f1, . . . , fs, one may fit j’s reliability as a linear combina- tion of the basis, i.e., rj (t) ≈Ps
i = 0 βifi (t), through regression analysis. The approximated rj (t) can separate the observed variations of j’s reliability into an explainable trend together with an error term. In studying variations of j’s reliability, it may suggest possible anomaly when the unexplainable error is far away from expected values. For regression analysis of interval-valued data and applications, readers may refer (Hu 2007, Hu 2008, Hu 2012, Hu and Hu 2020b).
COMPUTATIONAL EXPERIMENTS In our computational experiments, we investigate the impact of worker’s reliability on the overall quality of crowdsourced work.
Datasets, Methods, and Quality Measures In our experiments, we use four binary-classification benchmark datasets named Car, Income94, Sick and Vote, with known ground truth, in CEKA (Zhang et al. 2015) as testing datasets. Table 1 lists the size, number of attri- butes, and number of positive and negative instances of each dataset. The datasets are imbalanced except Income94. We use a portion of the test dataset as gold questions, and the rest as regular questions.
A virtual pool of one hundred workers is generated with various levels of reliability. Five inference making schemes are employed in our experiments. Three of them serve as baseline schemes. They are majority voting (MV), inte- rval majority voting (IMV), and preferred matching probability (PMP). Two reliability weighted schemes are weighted interval majority voting (WIMV) and weighted preferred matching probability (WPMP). Because the ground truth of each observation is known, we can form the confusion matrix in our experiment. We measure the overall quality with accuracy, precision, recall, and F1-score derived from confusion matrix.
Reliability Weighted Schemes Improve the Overall Quality Experiments on all four test datasets demonstrate that the reliability weigh- ted schemes can significantly improve the overall quality of crowd-sourced tasks. Figure 2 illustrates recall, precision, accuracy, and F1-score on the data- set Car with the five inference making schemes. In which, the horizontal
Figure 2: Comparison of performance on the test dataset car.
axis indicates the minimum confidence threshold for worker selection. Above each confidence threshold, ten workers are randomly selected from the pool for each run. The workers give IVLs on all questions; and the performance metrics are calculated based on the resulting confusion matrices. These scores are then averaged over forty runs to minimize any outliers due to the random factor. To utilize IVLs from workers with a low correctness (≤20%) and a high level of confidence (≥80%), we replace each of them with its difference from 1. The reliability weighted schemes WIMV and WPMP significantly outperform the three baseline schemes. Similar results are observed on the other three test datasets too. We do not include them in this paper to meet the page limit.
Applying Worker’s Reliability to Detect Anomaly Dynamically monitoring worker’s reliability can help us to detect anomaly and identify possible attackers. In our experiments, we set a fifth of the worker pool to
behave abnormally as designated as anomalous workers. Their labeling patterns on U and G are inconsistent. Applying the t- and F-tests for each j ∈J on Lj
G and Lj
G, we can detect them as illustrated in Fig. 3. In which, each worker is specified with his/her correctness and confidence as a light blue dot. A dark blue x indicates an identified anomalous worker. The probabilistic confidence level is 95% in the experiment.
Excluding Identified Attackers for Quality Crowdsourcing In our experiments, we further investigate the impact of excluding identified attackers. Fig. 4 compares the F1-score for each test dataset with or without excluding labels from identified attackers. The horizontal axis indicates the number of attackers present in the entire pool. By excluding identified atta- ckers, we have kept a near perfect F1-score for all test datasets. In contrast,
Dynamically Monitoring Crowd-Worker’s Reliability with Interval-Valued Labels 29
Figure 3: Detected possible attackers in labelling each dataset.Figure 4: Comparisons F1-score with or without excluding identified attackers.
the F1-score decreases rapidly as the number of attackers increases without attacker exclusion.
Impacts of Window Size on Monitoring Worker’s Reliability Monitoring j’s reliability dynamically, we should use a moving time window. Because a worker may not label instances all the time, we use a fixed num- ber of IVLs in a window as 5, 10, 50, 100, 150, and 200 in our experiment instead. We run the t- and F-tests for Lj
U. Fig. 5 illustrates the impact
G and Lj
30 Hu and Spurling
Figure 5: Impacts of window size on monitoring worker’s reliability.
of the number of IVLs in a window on the test dataset Car1. As the number of IVLs increases, the anomalies start to be singled out more. This is because the system remembers actions from previous IVLs. This is fine. We need enough samples in statistic and probabilistic tests.
CONCLUSION
Due to the open nature of crowdsourcing, crowd workers usually come with various levels of knowledge, social-economic backgrounds, and motivations. The reliability of crowd workers can impact the quality of crowdsourcing significantly. Using IVLs instead of common binary-valued ones, we can quantify the reliability of a particular worker in terms of correctness, con- fidence, stability, and predictability. Our computational experiments have demonstrated that applying worker’s reliability, one can improve the ove- rall quality of crowdsourcing through reliability weighted inference making, anomaly detection, and attacker exclution. A worker’s reliability often varies from time to time rather than constant. We need to monitor worker’s beha- vior and update worker’s reliability dynamically in practice. In this work, we treat IVLs from a worker j as a time series. Using IVLs within a forward moving time window, we can update and monitor j’s reliability dynami- cally. We can further analyze worker’s reliability with regression analysis of interval-valued sequence. Results of computational experiments indicate that we need, and we can monitor worker’s reliability dynamically.
ACKNOWLEDGMENT
This work is partially supported by the US National Science Foundation through the grant award NSF/OIA-1946391.
REFERENCES
Barbosa, N., and Chen, M. (2019) “Rehumanized crowdsourcing: a labeling framework addressing bias and ethics in machine learning”. Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems, pp. 1-12.
1Similar results are observed on other test datasets too.
Dynamically Monitoring Crowd-Worker’s Reliability with Interval-Valued Labels 31
Bi, W., Wang, L., Kwok, J., and Tu, Z. (2014) “Learning to predict from crowdsourced data”. UAI’14: Proceedings of the Thirtieth Conference on Uncertainty in Artificial Intelligence, pp. 82–91. Checco, A., Bates, J., and Demartini, G. (2020). Adversarial attacks on crowdsou-
rcing quality control. J. of Artificial Intelligence Research 67, pp. 375–408. Hu, C. (2008). Using interval function approximation to estimate uncertainty.
Advances in Soft Computing, vol 46. Springer, Berlin. Hu, C. and He, L. (2007). An application of interval methods to stock market forecasting. J. Reliable Computing 13, pp. 423–434. Hu, C. (2012). Interval function and its linear least-squares approximation. ACM
SNC ‘11: Proceedings of the 2011 International Workshop on Symbolic-Numeric Computation, pp. 16–23. Hu, C., and Hu ZH. (2020a). On statistics, probability, and entropy of intervalvalued datasets. Communications in Computer and Information Science 1239, pp. 407–421. Hu, C., and Hu ZH. (2020b). A Computational Study on the Entropy of Interval-
Valued Datasets from the Stock Market. Communications in Computer and Information Science, vol. 1239, pp. 422–435. Hu, C. Sheng, VS., Wu, N., and Wu, X. (2021). Managing uncertainties in crowdsourcing with interval-valued labeling. Lecture Notes in Networks and Systems 258, pp. 166–178. Qiu, L., et al (2016). CrowdSelect: Increasing accuracy of crowdsourcing tasks through behavior prediction and user selection. Proceedings of the 25th ACM International Conference on Information and Knowledge Management, pp. 539–548. Sheng, VS, and Zhang, J. (2019). Machine learning with crowdsourcing: A brief sum-
mary of past research and future directions. Proceedings of the AAAI Conference on Artificial Intelligence 33(1), pp. 9837–9843. Sheng, VS., Zhang, J., Bin, G., Wu, X. (2019) Majority voting and pairing with multiple noisy labeling. IEEE Trans. on Knowledge and Data Eng. 31(7), pp. 1355–1368. Spurling M., Hu C., Zhan, H., Sheng VS. (2021). Estimating crowd-worker’s relia-
bility with interval-valued labels to improve the quality of crowdsourced work. 2021 IEEE (SSCI), pp. 01–08. Spurling M., Hu C., Zhan, H., Sheng VS.(2022). Anomaly detection in crowdsourced work with interval-valued labels. Communications in Computer and Information Science, vol 1601. Springer, Cham. Tao, F., Jiang, L., Li, C. (2020). Label similarity-based weighted soft majority
voting and pairing for crowdsourcing. Knowledge and Information Systems 62, pp. 2521–2538. Wang, G., Wang, T., Zheng, H., and Zhao, B. (2014). Man vs. machine: practical adversarial detection of malicious crowdsourcing workers. Proc. of the 23rd USENIX Security Symposium, pp. 239–254. Zhang, J., Sheng, VS., Nicholson, B., and Wu, X. (2015). CEKA: A Tool for
Mining the Wisdom of Crowds. Journal of Machine Learning Research 16, pp. 2853–2858.
LISTED IN THE AHFE DIGITAL BOOKSTORE CATALOGUE
Explore the complete digital volume
Artificial Intelligence and Social Computing
Read this individual research paper free through the original publisher, or explore the full digital proceedings book.
ISBN 978-1-958651-48-3
Explore this ISBN in the AHFE Bookstore ↗Listed in the July 2026 bookstore catalogue; search by ISBN to confirm current availability · Digital book purchase optional