Measure one layer at a time
An end-to-end voice test combines microphone quality, room noise, accent, browser speech support, transcription, intent parsing, and map execution. When it fails, the number alone cannot explain why. VoiceGIS therefore evaluates its deterministic parser separately from speech recognition.
Each benchmark case supplies text and an expected structured result: intent, arguments, and relevant flags. The same corpus runs offline without a microphone, network connection, or model download.
Build a corpus around behaviours
A high score is only meaningful when the dataset represents the behaviours the library claims to support. The 538 cases are grouped so weak areas remain visible instead of being hidden inside one aggregate percentage.
- Navigation commands: cities, coordinates, fuzzy names, and alternative phrasing.
- Zoom and pan commands with direction, amount, and relative language.
- Layer visibility and aliases.
- Marker creation and relative locations.
- Reset, map switching, conversational phrases, and command chaining.
- Empty input, Unicode, malformed text, and other edge cases.
Make results reproducible
The harness is an ordinary command that produces a machine-readable result file and a non-zero exit when the enforced threshold or expected cases regress. CI can therefore treat parser behaviour like any other tested interface.
npm run evaluate
# Current evaluation summary
# navigation: 385 / 387
# overall: 504 / 538 (93.7%)Read the category scores, not only the headline
The current overall result is 93.7%, but coverage is uneven. Navigation performs strongly because it has the largest and most structured set. Conversational phrases and typo resilience are materially weaker. Those category results are more useful for prioritizing parser work than the overall score alone.
A benchmark also creates pressure to avoid tailoring rules only to known phrases. New cases should come from observed ambiguity classes and be reviewed for duplicates, unrealistic wording, and train-on-test behaviour.
What 93.7% does not mean
The figure is parser accuracy over a versioned text corpus. It is not word-error rate, microphone accuracy, real-user task completion, or a guarantee that every supported browser can transcribe speech. The public portfolio should keep that qualifier next to the number.
- Web Speech availability and privacy vary by browser.
- On-device Whisper adds model-download, CPU, and latency costs.
- A private server profile still requires someone to host and secure transcription.
- Correct parsing does not guarantee that an application should execute a destructive command without confirmation.