Handler Scores

Speed vs. Functionality

Scores are on scale of `0..100`. Higher is better.
Metrics algorithms completely made up.
See notes below for details.
Handler	Check box to see details Warnings	Check box to see details Benchmarks
slog/JSONHandler	98.67	97.08
snqk/meld	100.00	87.24
veqryn/dedup/Append	95.56	51.32
veqryn/dedup/Ignore	95.56	53.89
veqryn/dedup/Increment	95.56	51.62
veqryn/dedup/Overwrite	100.00	53.89

X Axis: `Warnings`

Numbers are scores for specific handler/warning level cases.
Scores are comprised of warnings counts per warning levels
mapped onto a 0..100 range. Higher is better.
Levels → ↓Handlers	Required	Implied	Suggested	Administrative	Check box to see detail score columns Score	by Data	Original
slog/JSONHandler	100.00	100.00	90.00	100.00	98.67	98.67	98.47
snqk/meld	100.00	100.00	100.00	100.00	100.00	100.00	100.00
veqryn/dedup/Append	100.00	100.00	100.00	33.33	95.56	95.56	98.47
veqryn/dedup/Ignore	100.00	100.00	100.00	33.33	95.56	95.56	98.47
veqryn/dedup/Increment	100.00	100.00	100.00	33.33	95.56	95.56	98.47
veqryn/dedup/Overwrite	100.00	100.00	100.00	100.00	100.00	100.00	100.00

Counts of *unique* warnings for handler / level
(not the total number of warning instances). Lower is better.
Levels → ↓Handlers	Required	Implied	Suggested	Administrative	Check box to see detail score columns Score	by Data	Original
slog/JSONHandler	0	0	1	0	98.67	98.67	98.47
snqk/meld	0	0	0	0	100.00	100.00	100.00
veqryn/dedup/Append	0	0	0	2	95.56	95.56	98.47
veqryn/dedup/Ignore	0	0	0	2	95.56	95.56	98.47
veqryn/dedup/Increment	0	0	0	2	95.56	95.56	98.47
veqryn/dedup/Overwrite	0	0	0	0	100.00	100.00	100.00

Y Axis: `Benchmarks`

Memory:

Numbers are scores for specific handler/test cases.
Scores are comprised of benchmark speed and memory usage data
mapped onto a `0..100` range. Higher is better.
Tests → ↓Handlers	Attributes	Big Group	Disabled	Key Values	Logging	Simple	Simple Source	With Attrs Attributes	With Attrs Key Values	With Attrs Simple	With Group Attributes	With Group Key Values	Check box to see detail score columns Score	by Test	by Data	Original
slog/JSONHandler	96.66	98.30	97.00	96.70	98.43	97.78	89.61	97.84	97.83	99.56	97.62	97.54	97.08	97.07	97.10	97.07
snqk/meld	83.45	78.15	94.63	82.66	87.93	97.39	89.23	84.16	84.13	90.34	88.75	88.79	87.24	87.47	87.01	87.47
veqryn/dedup/Append	46.54	9.24	94.63	45.89	46.94	84.06	84.21	36.22	35.55	44.02	52.26	51.91	51.32	52.62	50.01	52.62
veqryn/dedup/Ignore	45.98	8.93	94.25	46.15	47.17	84.27	84.28	50.54	50.49	44.23	52.52	52.61	53.89	55.12	52.67	55.12
veqryn/dedup/Increment	45.76	4.61	94.55	45.29	46.99	84.19	84.15	41.39	41.47	43.38	51.33	51.50	51.62	52.88	50.35	52.88
veqryn/dedup/Overwrite	46.36	10.01	94.58	46.43	47.07	84.25	84.15	50.09	49.99	44.32	51.95	52.36	53.89	55.13	52.66	55.13

Numbers represent microseconds per operation. Lower is better
Tests → ↓Handlers	Attributes	Big Group	Disabled	Key Values	Logging	Simple	Simple Source	With Attrs Attributes	With Attrs Key Values	With Attrs Simple	With Group Attributes	With Group Key Values	Check box to see detail score columns Score	by Data	by Test	Original
slog/JSONHandler	1.466	18.490	0.004	1.481	26.541	0.288	0.828	1.457	1.516	0.316	1.457	1.525	97.08	97.10	97.07	97.07
snqk/meld	2.503	70.026	0.004	2.777	48.150	0.312	0.883	3.553	3.621	1.720	2.556	2.585	87.24	87.01	87.47	87.47
veqryn/dedup/Append	5.689	294.533	0.004	5.853	129.488	0.503	1.062	10.926	11.315	5.526	7.110	7.303	51.32	50.01	52.62	52.62
veqryn/dedup/Ignore	5.881	296.388	0.004	5.761	127.618	0.490	1.052	8.416	8.491	5.454	7.006	7.028	53.89	52.67	55.12	55.12
veqryn/dedup/Increment	5.950	314.640	0.004	6.051	129.207	0.495	1.070	11.048	11.056	5.744	7.465	7.452	51.62	50.35	52.88	52.88
veqryn/dedup/Overwrite	5.753	289.786	0.004	5.668	128.396	0.492	1.070	8.422	8.524	5.424	7.231	7.126	53.89	52.66	55.13	55.13

Numbers represent number of memory allocations per operation. Lower is better.
Tests → ↓Handlers	Attributes	Big Group	Disabled	Key Values	Logging	Simple	Simple Source	With Attrs Attributes	With Attrs Key Values	With Attrs Simple	With Group Attributes	With Group Key Values	Check box to see detail score columns Score	by Data	by Test	Original
slog/JSONHandler	6	1	0	6	0	0	6	6	6	0	6	6	97.08	97.10	97.07	97.07
snqk/meld	31	1,469	0	31	561	0	6	38	38	8	34	34	87.24	87.01	87.47	87.47
veqryn/dedup/Append	41	2,068	0	41	867	6	12	100	100	39	50	50	51.32	50.01	52.62	52.62
veqryn/dedup/Ignore	41	2,068	0	41	867	6	12	61	61	39	50	50	53.89	52.67	55.12	55.12
veqryn/dedup/Increment	41	2,177	0	41	867	6	12	74	74	39	50	50	51.62	50.35	52.88	52.88
veqryn/dedup/Overwrite	41	2,068	0	41	867	6	12	63	63	39	50	50	53.89	52.66	55.13	55.13

Numbers represent bytes of memory allocated per operation. Lower is better.
Tests → ↓Handlers	Attributes	Big Group	Disabled	Key Values	Logging	Simple	Simple Source	With Attrs Attributes	With Attrs Key Values	With Attrs Simple	With Group Attributes	With Group Key Values	Check box to see detail score columns Score	by Data	by Test	Original
slog/JSONHandler	472	48	0	472	0	0	568	472	472	0	472	472	97.08	97.10	97.07	97.07
snqk/meld	2,186	73,090	0	2,186	30,214	0	568	3,363	3,363	1,001	2,074	2,074	87.24	87.01	87.47	87.47
veqryn/dedup/Append	9,724	659,978	0	9,724	224,776	280	848	16,464	16,470	9,050	13,270	13,273	51.32	50.01	52.62	52.62
veqryn/dedup/Ignore	9,725	660,149	0	9,726	224,813	280	848	14,074	14,074	9,051	13,274	13,270	53.89	52.67	55.12	55.12
veqryn/dedup/Increment	9,728	675,194	0	9,727	224,684	280	848	15,323	15,323	9,051	13,284	13,289	51.62	50.35	52.88	52.88
veqryn/dedup/Overwrite	9,722	660,233	0	9,726	224,837	280	848	14,171	14,170	9,051	13,272	13,272	53.89	52.66	55.13	55.13

Score Visualization

	← Chart Size →	Top Right:

Higher numbers are better on both axes. The "good" zone is the upper right and the "bad" zone is the lower left.
The top is fast, the bottom is slow. Left is more warnings, right is less.

Scoring Algorithms

The algorithms behind the scores shown on this page are somewhat arbitrary. The original scoring algorithm (Default) was deemed "good enough", but later work has focused on enabling multiple scoring algorithms. These can be found on the Home page or in the Scoring drop-down in the upper right section of every page.

Algorithms are implemented by "scorekeepers". Each scorekeeper is specified by the two axes shown in the scoring chart. Each axis interprets test data according to its own algorithm.

The current scorekeeper and axis algorithms are described below:

Score Keeper: +Dedup

The Dedup scoring algorithm is basically the same as the Default algorithm. Only the handlers shown in the chart is changed.

The chart compares the behavior of the slog/JSONHandler handler to available wrappers that de-duplicate fields in the result. As of 2024-08-21 both such handlers are wrappers around preexisting log/slog handlers, in this case slog/JSONHandler.

X Axis: Warnings

The X-axis for the Default scoring chart shows the score derived from verification warnings.

The score is calculated using the score weights shown to the right which are applied to the warning levels during calculation.

Handlers are scored based on how few warnings are generated. Warnings are worth different amounts depending on their warning level. The weights applied during this process are shown on the right.

Source Data

Each handler results in a lot of verification test output:

Warnings for slog/JSONHandler:
  Suggested
     2 [Duplicates] Duplicate field(s) found
         TestAttributeDuplicate: map[alpha:2 charlie:3]
         TestAttributeWithDuplicate: map[alpha:2 charlie:3]

The Suggested line is an example of a warning "level" (warnings are grouped into levels on the warning page). In this example there are two instances of the Duplicates warning.

Warnings Algorithm

Scoring is done for all handlers at the same time:

for each handler
    score starts at zero
    for each warning level
        for each warning in level
            if warning shows up during testing
                score = score + weight(level) * len(warnings)
                adjust score to range of
                    zero to maximum possible number of warnings

Where the weight(level) comes from the predefined table shown above and to the right.

The scores for each handler are then divided by the maximum possible number of warnings that any handler might receive (if it were really awful) and that number is subtracted from 100.0. This results in a number from 0.0 (awful, all warnings logged) to 100.0 (no warnings at all). That number is stored for use and displayed on this page and on each handler page.

Note that most scores are above ~40 as it is difficult to throw all the warnings.

Scores

Multiple scores are generated for each handler. The main (or "default") score is shown in the data tables with the column header Score with an associated checkbox. The checkbox can be used to show several other "score" columns, as follows:

Default (Score)
This is the score that is shown in the overall chart at the top of the page in the column labeled Warnings. The default score is the same as the By Data score.
By Data
This score is calculated by rolling up scores calculated per warning level.
Original
This is the "original" score which has been overtaken by newer code. The Original score is within 5% of by Data value.

Level	Weight
Required	8
Implied	4
Suggested	2
Administrative	1

Y Axis: Benchmarks

The Y-axis for the Default scoring chart shows the score derived from running benchmarks.

The score is calculated using the score weights shown to the right which are applied to the several specific benchmark result values.

Handler benchmarks are scored based on several metrics on each of various tests. Metrics are worth different amounts depending on what they are. The weights applied during this process are shown on the right.

Source Data

Each combination of handler and test results in a single line of test output:

BenchmarkMadkinsFlash/BenchmarkSimple-8  3547497  327.1 ns/op  284.33 MB/s  24 B/op  1 allocs/op

From this line we get:

the handler name (BenchmarkMadkinsFlash),
the test name (BenchmarkSimple),
the number of test runs (3547497),
nanoseconds per operation (327.1 ns/op),
memory bytes allocated per operation (24 B/op),
separate memory allocations per operation (1 allocs/op), and
estimated logging throughput per second (284.33 MB/s).

Benchmark Algorithm

For each handler/test combination (single line or test results) we use one or more of the following three data items:

nanoseconds per operation,
memory bytes allocated per operation, and
separate memory allocations per operation.

These three items are combined over two steps. First the test value ranges are acquired:

for each test
    for each handler
        for each of the three results described above
            track the highest and lowest value for the test over all handlers

Then the test scores are calculated (this is the Original calculation):

for each handler
    for each test
        scorePerTest starts at zero
        for each of the three results described above
            convert the value to a fraction of
                the range of values for the test from the previous step
            scorePerTest = scorePerTest + weight(result) * 100.0 * the fraction
        scorePerTest /= sum of weight(result)
    scorePerHandler = average of scorePerTest for handler

Where the weight(result) comes from the predefined table shown above and to the right. There is currently no weighting by test, all tests are considered equal.

Scores

Default (Score)
This is the score that is shown in the overall chart at the top of the page in the column labeled Benchmarks. The default score is the average of the By Test and By Data scores.
By Test
This score is calculated by rolling up scores calculated per test.
By Data
This score is calculated by rolling up scores calculated per data item.
Original
This is the "original" score which has been overtaken by newer code. The Original score is within 5% of by Data value.

Data	Weight
Nanoseconds	3
Alloc Bytes	2
Allocations	1