Task success
Did the AI reach a technically correct, useful result?
The ITChat method
Measure the work.
Not the claim.
We give IT tools the same practical jobs and evaluate what happens from prompt to outcome. The scorecard covers correctness, time saved, safety, and whether the AI recognizes its limits.
Context and evidence
Did it use the device, ticket, and policy context—or fill gaps with guesses?
Safety and control
Did it respect permissions, approval gates, and the impact of an action?
Technician effort
How much time, correction, and rework remained for the human?
Escalation judgment
Did it ask for more information or involve a person at the right time?
Repeatability
Did performance hold up across repeated runs and small changes in context?
The first benchmark
AI for endpoint
and service desk work.
We are starting with the everyday tasks where AI promises to save technicians time—and where a bad answer or careless action can create real work.
Investigate
Find the cause
Use device state, alerts, and recent activity to explain what may be wrong and what evidence is still needed.
Remediate
Recommend a safe fix
Propose a useful next step, explain the risk, and preserve the approval a technician should have.
Automate
Build a package or script
Create a reviewable deployment or remediation plan that handles constraints and missing information.
Support
Move the ticket forward
Summarize the issue, suggest a response, and surface a resolution grounded in the available context.
Initial product landscape: Atera, NinjaOne, PDQ Connect, and ManageEngine. Action1 is on the watchlist; it will be directly scored when a comparable AI capability is confirmed. Product versions, access, and test dates will be disclosed in every report.
Built with the people doing the work
Useful to operators.
Credible to buyers.
ITChat combines controlled tests with experienced practitioners’ judgment. We are recruiting a small pilot panel and speaking with IT leaders who need better evidence for AI decisions.
For IT practitioners
Help define realistic scenarios and judge whether an AI answer would work in the field. Participation is structured, time-bounded, and compensated for research sessions.
- Bring the tasks you repeat every week
- Help identify unsafe or misleading answers
- Choose separately how your input can be used
For IT leaders
Use task-level evidence to compare AI workflows, set rollout expectations, and understand where technicians still need to stay in control.
- See what the tool completed and what it missed
- Compare human effort alongside task quality
- Get a clear record of setup, method, and limits
Trust is part of the test
Clear methods.
Visible limits.
Every result should tell you what was tested, how it was scored, and what the test cannot establish. ITChat will publish its method and disclose funding relationships.
Founder disclosure: ITChat’s founder is also CEO of PDQ, whose product may be evaluated. Any PDQ results will use the same published protocol, independent scoring, and prominent disclosure. No nonpublic PDQ or customer data will be used.
01
Same task intent
Scenarios and expected outcomes are defined before product runs.
02
Human review
Practitioners and evaluators check whether outputs are correct and safe.
03
Show the details
Reports identify feature access, product version, date, sample size, and scoring limits.
04
No bought rankings
Funding can support research; it cannot buy a score or control a conclusion.
Early pilot
Help us test what AI can do.
We’re looking for experienced IT practitioners to shape the first task set and IT leaders who want to see the results. The pilot is being formed now.