Documenting machine learning (ML) experiments thoroughly and clearly is critical for reproducibility, collaboration, and progress tracking. Using prompt chains—a series of structured, purposeful prompts—can systematically guide the documentation process, ensuring no important detail is missed and that insights are captured consistently. Here’s a comprehensive approach to prompt chains for better documentation of ML experiments:
1. Experiment Overview Prompt Chain
-
What is the main objective or hypothesis of this experiment?
(Clarify the problem statement or research question.) -
What is the expected outcome or metric improvement target?
(Set clear goals for success or benchmarks.) -
What is the experiment’s context within the broader project or pipeline?
(Explain how this experiment fits into the overall workflow.)
2. Dataset and Data Processing Prompt Chain
-
Which dataset(s) are being used? Include versions and sources.
-
How was the data split (train/validation/test)?
-
What preprocessing or feature engineering steps were applied?
(Normalization, augmentation, missing value handling, encoding, etc.) -
Are there any data quality issues or anomalies noted?
3. Model Architecture and Configuration Prompt Chain
-
What model architecture or algorithm is used?
(Include model type, layers, parameters, or any relevant design choices.) -
What are the hyperparameters and their chosen values?
(Learning rate, batch size, number of epochs, optimizer, etc.) -
Were any pre-trained models or transfer learning techniques used?
-
Are there any modifications or custom components in the model?
4. Training Details Prompt Chain
-
What hardware and software environment were used?
(Specify GPUs, CPUs, OS, framework versions.) -
What was the training procedure?
(Epochs, early stopping, checkpointing, loss functions.) -
Were there any issues or anomalies during training?
-
How was model validation conducted?
5. Evaluation and Metrics Prompt Chain
-
What metrics were used to evaluate the model?
(Accuracy, F1, precision/recall, loss values, AUC, etc.) -
What were the final evaluation results on validation and test sets?
-
How do results compare to baseline or previous experiments?
-
Are there any observed biases or limitations in the evaluation?
6. Interpretability and Insights Prompt Chain
-
What insights were gained from the model or results?
-
Were any feature importances or model explanations generated?
-
Are there notable patterns, anomalies, or failures in predictions?
7. Reproducibility and Next Steps Prompt Chain
-
Are all scripts, configurations, and datasets version-controlled and accessible?
-
Is the experiment fully reproducible with the provided documentation?
-
What are the immediate next steps or hypotheses to test?
-
What improvements or variations will be tried next?
8. Reflection and Lessons Learned Prompt Chain
-
What worked well in this experiment?
-
What challenges or failures were encountered?
-
How could the experiment design or documentation be improved?
-
Are there any risks or caveats to consider for deployment?
Benefits of Using Prompt Chains for Documentation
-
Consistency: Ensures every experiment is documented in the same structured way.
-
Completeness: Avoids missing crucial details that could hinder reproducibility.
-
Clarity: Helps collaborators quickly understand what was done and why.
-
Insightfulness: Encourages reflection on results and planning for future work.
By systematically answering these prompts for each ML experiment, teams can build a rich, searchable knowledge base that accelerates development and innovation. Prompt chains can be integrated into notebooks, lab logs, or specialized experiment tracking tools to streamline the documentation workflow.