Vision - Duplicate Images

Keep Exploring!!!

Python + Data Pipelines

Data fetched from multiple sources

Data Integrity purity test

Lessons

ETL vs ELT

DAG / Pre-requisites, Remember order of sequence

• Many ordering would be there
• CDC will have issues

Lesson

Longer bad data more cost to pay

Backward compatible support schema

Keep Exploring!!!

Pytube error - RegexMatchError: get_throttling_function_name: could not find match for multiple

Pytube error - RegexMatchError: get_throttling_function_name: could not find match for multiple

This error locates to file C:\ProgramData\Anaconda3\envs\tensorflow\lib\site-packages\pytube\cipher.py

Fix in link

One line Fix but a lot of misdirections / useless pointers

Fixing the issue in the least time / right debugging matters.

Keep Exploring!!!

Python - Concurrency - Examples

Keep Exploring!!!

AWS Lambda Basics

 Basic Example flow

1. Basic Python Code

2. Create a lambda function

3. Replace function insider handler - change handler - entry point

4. Test the handler

5. Add Trigger to invoker from outside / Gateway

6. Test from Browser

Keep Exploring

Ref - Link

Performance profiling

Happy Learning!!!

Interesting observations tesseract

While extracting digits from analog meters below two links we use to get the values

Lesson #1 - Setting the path to a folder vs complete executable, Minor thing took a while since not using it often

Ref - Link

Lesson #2 - Very useful for different situations on how it can be interpreted, 11 worked best. 6 was ok

Ref - Link

Keep Exploring!!!

OpenCV Experiments

Experiment with

• Logging
• Threads
• Error Handling
• Writing text on Frame with grey background

Keep Exploring!!!

Python coding review tools - Docker, Kubernetes

Good read on code quality Tools - Link

Picked and checked on below tools for example project

• Bandit - security issues
• Pylint - coding standard 
• pycodestyle - code style 

Download Project - Link

Result #1

Result #2

Result #3

This CI / CD 8th Lab video  was motivational to explore these tools. Need to re-read this session again.

Tool Links

pylint

bandit

pycodestyle

code - link

Todo List

Good Reads

• Exploring Shap - Link1, Link2
• Python Best Practices for a New Project in 2021 - Link1
• ML-Ops_ModelDeployment_k8s - Link1

Keep Exploring!!!

Simple Utility - File Renumbering

 

Script to renumber files in all subdirectories. Bookmarking it for future use.

Keep Learning!!!

Examples Flask vs Fast API

Recently FastAPI I could see more posts/recommendations compared to flask API in a performance context. A basic example of implementation with flask vs Fast API. The format  /syntax, request differences you can spot by comparing them.

CI / CD

DevOps with Github

Makefile

Happy Learning!!!

Day #336 - Image Encoding / Decoding Python

Happy Learning!!!

Learning Notes - Convex Optimization in Python with CVXPY

Key Notes
Convex Optimization problem

• Decision variable to solve
• Objective Function
• Inequality Functions
• Linear equality constraints
• All functions are convex
• Convexity - Positive curvature, curve upwards like parabola

Useful in below fields

• Applies in various fields
• All ML algos based on Convex Optimization
• SpaceX landing Convex Optimization

How to Solve ?

• Using Solvers for particular form of problem
• Linear program, quadratic problem, second order con program

Convexity Verification

• Using Disciplined convex programming
• Determine curvature of every node in expression tree

cvxpy

• Developed in python
• Modelling framework
• Parameter assignment
• Simulation visualization
• Energy Management (Load Demand, Battery Charge / Discharge / Price margin / Load)

Keep Thinking!!!

Learning Notes - Optimization

Key Notes

• Three parameters for Linear problems
• Decision variables (Number of quantities to decide)
• Objective (Minimize / Maximize profit / time)
• Constraints (Time / Resources)
• Identify these three parameters for your problem
• Leverage existing Packages

How it solves

• Decision variable multiplied by cost, subject to the constraint
• Different modeling frameworks

Fundamentals (Link)

• Convex optimization involves minimizing a convex objective function
• Linear programming is a special case of convex optimization where the objective function is linear and the constraints consist of linear equalities and inequalities
• Linear programming is a special case of convex programming, in which the objective function is a linear

Link, Link1

• Optimization is when you search for variables that attain a global maximum or minimum of some function
• Convex optimization is a subset of optimization where the functions you work with are "convex" which just means "bowl shaped". This makes the search for maxima and minima easier since you can just " walk " on the surface of the bowl in the direction with the greatest slope to get there.

Keep Thinking!!!