Offer summary

Qualifications:

Advanced degree in Computer Science, Software Engineering, or a related field., 5+ years of extensive experience in software development across multiple programming languages., Strong analytical and decision-making skills with the ability to evaluate code quality and efficiency., Excellent written and verbal communication skills, particularly in technical writing..

Key responsabilities:

Critically analyze and evaluate code responses generated by LLMs across various programming languages.

Select the most appropriate and efficient code solutions from multiple LLM-generated options.

Develop coding demonstrations to illustrate quality benchmarks for AI-generated code.

Collaborate with the AI research team to identify areas for improvement in LLM's coding capabilities.

Job description

Pay rate range - $50/hr. to $55/hr. on W2
100% Remote
Part-time - Only 10 hours per week

Job Description:
As an LLM Evaluation Expert specializing in Coding, you will play a crucial role in assessing and improving our language models' coding capabilities.
Your expertise will be instrumental in evaluating LLM-generated code responses, making high-level judgments, and setting the standard for what constitutes excellent AI-assisted coding.

Key Responsibilities:
• Critically analyze and evaluate code responses generated by our LLMs across various programming languages and paradigms
• Exercise expert judgment to select the most appropriate and efficient code solutions from multiple LLM-generated options
• Make informed decisions on behalf of our customers, ensuring that selected code meets industry standards, best practices, and specific client needs
• Develop and write coding demonstrations to illustrate "what good looks like" in AI-generated code, setting benchmarks for quality and efficiency
• Provide detailed feedback and explanations for your evaluations, helping to refine and improve the LLM's understanding and output
• Collaborate with the AI research team to identify areas for improvement in the LLM's coding capabilities
• Stay abreast of the latest developments in software engineering, coding standards, and AI to ensure our evaluations remain cutting-edge

Required Qualifications:
• Advanced degree in Computer Science, Software Engineering, or a related field
• Extensive experience (5+ years) in software development across multiple programming languages and paradigms
• Demonstrated ability to critically evaluate code quality, efficiency, and adherence to best practices
• Strong analytical and decision-making skills, with the ability to make complex judgments under ambiguous circumstances
• Excellent written and verbal communication skills, with the ability to explain technical concepts clearly
• Experience in technical writing, particularly in creating coding examples or tutorials

Preferred Qualifications:
• Previous experience working with or evaluating AI systems, particularly in the context of code generation
• Familiarity with a wide range of software development methodologies and architectural patterns
• Understanding of machine learning concepts, particularly as they apply to natural language processing and code generation
• Experience in creating or contributing to coding standards or style guides
This role requires a unique blend of technical expertise, critical thinking, and communication skills. You will be the bridge between advanced AI technology and practical, real-world coding applications. Your work will directly influence the development of next-generation AI coding assistants, shaping the future of software development.
If you're passionate about code quality, have a keen eye for detail, and are excited about the potential of AI in software engineering, we encourage you to apply for this pivotal role at AGI Data Services.

Required profile