Separate parsing from speech recognition
An end-to-end voice test mixes microphone quality, room noise, accent, browser support, transcription, parsing, and map execution. When one number covers all of that, a failure does not tell me which layer broke. VoiceGIS therefore tests its rule-based parser separately from speech recognition.
Each case provides text and the structured result I expect: an intent, its arguments, and any relevant flags. The corpus runs offline without a microphone, network connection, or model download.
Split the corpus by behaviour
An overall score is useful only if I can see what produced it. I group the 538 cases by behaviour so a strong navigation result cannot hide weak typo handling or conversational language.
- Navigation commands: cities, coordinates, fuzzy names, and alternative phrasing.
- Zoom and pan commands with direction, amount, and relative language.
- Layer visibility and aliases.
- Marker creation and relative locations.
- Reset, map switching, conversational phrases, and command chaining.
- Empty input, Unicode, malformed text, and other edge cases.
Put the benchmark in CI
The harness is an ordinary command. It writes a machine-readable result and exits non-zero when an enforced threshold or expected case regresses, so CI can treat parser behaviour like any other tested interface.
npm run evaluate
# Current evaluation summary
# navigation: 385 / 387
# overall: 504 / 538 (93.7%)The overall score hides weak categories
The current result is 93.7%, but the categories are uneven. Navigation is strong because it has the largest and most structured set. Conversational phrases and typo resilience are materially weaker. Those gaps tell me where to work next; the aggregate score does not.
There is another trap: tuning rules only for phrases already in the file. I add cases from observed ambiguity classes and review them for duplicates, unrealistic wording, and train-on-test behaviour.
What 93.7% does—and does not—measure
The figure is parser accuracy over a versioned text corpus. That qualifier belongs beside the number: it is not word-error rate, microphone accuracy, real-user task completion, or a guarantee that every browser can transcribe speech.
- Web Speech availability and privacy vary by browser.
- On-device Whisper adds model-download, CPU, and latency costs.
- A private server profile still requires someone to host and secure transcription.
- Correct parsing does not guarantee that an application should execute a destructive command without confirmation.