🤝 Contributing to RatCrawler

Help us improve RatCrawler and make it even better

🚀 Getting Started

First steps to contribute to RatCrawler

1. Fork & Clone

Fork the Repository

# Click the "Fork" button on GitHub
# This creates your own copy of the repository

Clone Your Fork

git clone https://github.com/swadhinbiswas/ratcrawler.git
cd ratcrawler

Set Up Upstream

git remote add upstream https://github.com/swadhinbiswas/ratcrawler.git
git fetch upstream

2. Development Setup

Python Setup

# Install Python dependencies
pip install -r requirements.txt

# Install development dependencies
pip install pytest black flake8 mypy

Rust Setup

# Install Rust (if not already installed)
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh

# Build the project
cd rust_version
cargo build

Verify Setup

# Test Python installation
python3 main.py --help

# Test Rust installation
cargo test

🔄 Development Workflow

How to work on RatCrawler features and fixes

Create Feature Branch

Always work on a separate branch for your changes 

# Create and switch to a new branch
git checkout -b feature/your-feature-name

# Or for bug fixes
git checkout -b fix/issue-description

# Or for documentation
git checkout -b docs/update-readme

# Example branches:
# feature/add-google-trends-integration
# fix/crawler-timeout-issue
# docs/add-api-examples
# refactor/improve-error-handling

Make Your Changes

Implement your feature or fix

Code Standards

  • • Follow PEP 8 for Python code
  • • Use rustfmt for Rust code
  • • Write clear, descriptive commit messages
  • • Add comments for complex logic
  • • Keep functions focused and small

Testing

  • • Write unit tests for new features
  • • Test edge cases and error conditions
  • • Ensure existing tests still pass
  • • Test both Python and Rust versions

Commit & Push Changes

Save your work and push to your fork 

# Check what files you've changed
git status

# Add your changes
git add .

# Or add specific files
git add filename.py
git add src/new_module.rs

# Commit with a descriptive message
git commit -m "feat: add Google Trends integration

- Add GoogleTrends class for trend analysis
- Implement trend data fetching
- Add trend visualization
- Update documentation

Closes #123"

# Push to your fork
git push origin feature/your-feature-name

Create Pull Request

Submit your changes for review

1. Go to your fork on GitHub
2. Click "Compare & pull request"
3. Fill out the pull request template:
   - Title: Brief description of changes
   - Description: Detailed explanation
   - Link any related issues
4. Click "Create pull request"

Pull Request Template:
```
## Description
Brief description of what this PR does.

## Changes Made
- Change 1
- Change 2
- Change 3

## Testing
- [ ] Unit tests pass
- [ ] Integration tests pass
- [ ] Manual testing completed

## Screenshots (if applicable)
Add screenshots of new features or UI changes.

Fixes #123
```

📋 Contributing Guidelines

Rules and best practices for contributions

🎨 Code Style

Python

  • • Follow PEP 8 style guide
  • • Use type hints for function parameters
  • • Maximum line length: 88 characters
  • • Use docstrings for all public functions
  • • Use snake_case for variables and functions

Rust

  • • Use rustfmt for code formatting
  • • Follow Rust naming conventions
  • • Add documentation comments for public APIs
  • • Handle errors properly with Result types
  • • Write comprehensive unit tests

📝 Commit Messages

Format

type(scope): description [optional body] [optional footer]

Types

  • feat: New feature
  • fix: Bug fix
  • docs: Documentation
  • style: Code style changes
  • refactor: Code refactoring
  • test: Adding tests
  • chore: Maintenance

Examples

  • feat: add backlink analysis
  • fix: handle timeout errors
  • docs: update API documentation
  • test: add crawler unit tests

🧪 Testing Guidelines

Python Testing

# Run all tests
pytest

# Run specific test file
pytest test_crawler.py

# Run with coverage
pytest --cov=ratcrawler

# Run tests in verbose mode
pytest -v

# Run specific test function
pytest test_crawler.py::test_basic_crawl

Rust Testing

# Run all tests
cargo test

# Run specific test
cargo test test_crawler

# Run with verbose output
cargo test -- --nocapture

# Run benchmarks
cargo bench

# Run tests in release mode
cargo test --release

📚 Guidelines

Important guidelines for contributors

Issue Reporting

  • • Use issue templates when available
  • • Provide clear reproduction steps
  • • Include error messages and logs
  • • Specify your environment (OS, Python/Rust version)
  • • Check for existing issues first

Feature Requests

  • • Clearly describe the feature
  • • Explain the use case and benefits
  • • Consider backward compatibility
  • • Provide mockups if applicable
  • • Discuss implementation approach

Code Review

  • • Review code for functionality
  • • Check code style and conventions
  • • Verify tests are included
  • • Ensure documentation is updated
  • • Test the changes locally

Documentation

  • • Update README for new features
  • • Add docstrings to new functions
  • • Update API documentation
  • • Include code examples
  • • Document configuration options

Security

  • • Never commit sensitive data
  • • Use environment variables for secrets
  • • Validate all inputs
  • • Follow secure coding practices
  • • Report security issues privately

Community

  • • Be respectful and inclusive
  • • Help other contributors
  • • Participate in discussions
  • • Share your knowledge
  • • Celebrate successes together

🆘 Getting Help

Where to Ask Questions

  • • GitHub Issues: For bugs and feature requests
  • • GitHub Discussions: For general questions
  • • Documentation: Check the docs first
  • • Existing Issues: Search before creating new ones

What to Include

  • • Clear description of the problem
  • • Steps to reproduce (if applicable)
  • • Expected vs actual behavior
  • • Your environment (OS, versions)
  • • Error messages and stack traces