Sunday Perspective - Data Scientist

Happy Learning!!!

Day #107 - Data Analysis and Exploration

	import pandas as pd
	from datetime import datetime
	import csv
	import matplotlib.pyplot as plt
	import matplotlib.dates as mdates
	import seaborn as sns
	import numpy as np
	import matplotlib.cbook as cbook
	df = pd.read_csv('Booth531113.csv',header=0)
	print (df)
	df['Date'] = df['Date'].map(lambda x: datetime.strptime(str(x), '%m/%d/%Y'))
	x = df['Date']
	y = df['Temp']
	y1 = df['Mains']
	y2 = df['MCU']
	y3 = df['Freezer']
	plt.plot(x, y)
	plt.plot(x, y1)
	plt.plot(x, y2)
	plt.plot(x, y3)
	plt.gca().set_color_cycle(['red', 'green', 'blue', 'yellow'])
	plt.legend(['Temp', 'Mains', 'MCU', 'Freezer'], loc='upper left')
	plt.title('113')
	plt.show()

view raw plotdata.py hosted with ❤ by GitHub

Happy Learning!!!

Day #106 - OpenCV for Python3

Finally installed OpenCV for python3 following steps in link 

Anaconda 3 Distribution works fine perfectly!

pip install opencv-python

Happy Learning!!!

Day #105 - Ensemble Tips and Tricks

Diversity based on Algorithms

• 2~3 gradient boosted trees (lightgb, xgboost, catboost)
• Neural networks (Keras, Pytorch)
• 1~2 Extra trees (Random Forest)
• 1-2 knn models

Diversity based on inputdata

• Categorical features (one hot, label encoding, target encoding)
• Numerical features (Outliers, binning, derivatives, percentiles)
• Interactions (col1*/+-col2),groupby,unsupervised

Subsequent level tips

• GDM with depth 3
• Linear models with high regularization
• Extra trees

Happy Learning!!!

Smart City Analytics

A picture is worth thousand words. This picture tweeted World Economic Forum is worth million words. A great example of smart city with the list of use cases. Every use case has implied analytics on top of it. A summary of it listed together

These three robot #cities could be the models of the future https://t.co/IcWi08YXcQ #automation pic.twitter.com/5UZr8s5gY1

— World Economic Forum (@wef) April 11, 2018

Area	Concept	Explanation	Learning	Analytics Role
Buildings	Green Building	Roof Top garden, Asorb CO2 produce Oxygen	Architecture to reduce Co2, Better lighting reduce energy consumption
Buildings	Building management	Automation and Optimization of Hearing, Lighting, ventilation		Analytics to predict energy consumption, cooler maintenance, predict failures, Anamoly detection
Buildings	Fire safety	Intelligent extinguishing customized based on design / plan / materials / goods in it	based on items kept decide on type and quantity and type of extinguisher
Environment	Permeter Access Control	Access, Moniutoring CCTV, Intruder detection		Video Analytics, Face Recognition, Eye Detection
Environment	RoofTop Wind Turbine	For high rise buildings	Renewable energy generation	Monitor the turbine performance, Identify any anamoly in device operations to predict failures, Forecast on performance or energy generation based on trends and seasons
Environment	Air Pollution Control	Control Co2 Emissions of Factories		Predict growing CO2 Emissions based on increasing number of factories, vehicles
Environment	Building Integrated Photovoltaics	Solar Panel integrated into building fabric to replace conventional materials	Renewable energy generation	Device Monitoring, Predictive maintenance
Environment / Infrastructure	Smart Grid	Energy Consumption and Monitoring		Predict , Forecast Energy needs
Environment/Buildings	Chemical Leakage Detection	Detecting Leakages / Wastes of factories in rivers	Identify / Penalize the faulty / corrupt ones	Water Quality Analysis to detect and identify the waste induction points / points of failure
Environment/Buildings	Real time Traffic updates	Instant Traffic updates sent to smart phones to help route planning		Already google traffic does crowd sourcing of Data and predicts traffic updates / delays
Infrastructure	Vertical Axis Wind Turbines	Vertical twisted wind turbines	Similar to solar power lights across city with renewable energy sources
Infrastructure	Structural health		Monitor building infrastructure and condition	Analytics to classify, eliminate old / low quality buildings
Life	Waste Management	Monitoring waste levels to optimize / refuse collection routes	Efficient garbage management	Video Analytics for garbage classification and segmentation
Life	Smart Parking	Parking Monitoring and management		Planning, Video Analytics to identify empty parking slots
Life	Earthquake Detection			Analytics to find anamoly in readings to predict EarthQuarke
Utilities	Potable Water Monitoring			Monitor ground water levels, contamination, forecast the availability
Utilities	Wifi	In Metro / Public Places		Classify, cluster usage based on gender / age group / economic levels
Utilities	Water Leakage Detection		Sensors to identify / monitor leaks in water supply
Utilities	Landslide prevention		Soil monitoring for early detection and prevention
Utilities	Fast Lane / Smart Signals		Optimize / divert based on real time traffic situations
Transport	Electrical Transport		Renewable enegy based transport systems

Happy Learning!!!

Day #104 - Working with DropBox

Python script to upload and download from dropbox. Install the dropbox package, get access token by signing up for dropbox. Create an app in dropbox console - Create an App on dropbox console (https://www.dropbox.com/developers/apps) and obtain access token

	#Upload to drop box
	import dropbox
	access_token = 'XXXX'
	#upload from local to dropbox
	file_from = 'Pics.jpg' #local file path
	file_to = '/Pics.jpg' # dropbox path
	def upload_file(file_from, file_to):
	dbx = dropbox.Dropbox(access_token)
	f = open(file_from, 'rb')
	dbx.files_upload(f.read(), file_to)
	upload_file(file_from,file_to)
	#Download from drop box to local
	dbx = dropbox.Dropbox(access_token)
	dbx.files_download_to_file('download1.jpg', '/'+ file_from)

view raw dropbox.py hosted with ❤ by GitHub

Happy Learning!!!

Day #103 - Hosting a flask Rest API

This post is on hosting a flask / rest API. This requires flask package. In Python 3.4 It was straight forward implementation

To execute run the code in python console - python example_gist.py

	import flask
	from flask import Flask, request
	from shutil import copyfile
	import subprocess
	import base64
	#Install flask
	app = flask.Flask(__name__)
	app.config["DEBUG"] = True
	@app.route('/UploadImagedata')
	def UploadImagedata():
	user = request.args.get('user')
	searchimagedata = str(request.args.get('imagedata'))
	searchimagedata = searchimagedata.strip()
	searchimagedata = searchimagedata.lstrip()
	searchimagedata = searchimagedata.encode()
	searchimagedata = searchimagedata.replace(' ','+')
	fh = open("E:\\MobileTestImage2.jpg", "wb")
	fh.write(searchimagedata.decode('base64'))
	fh.close()
	return searchimagedata
	#http://localhost:5000/UploadImagedata?user=siva&imagedata=XXXX
	@app.route('/UploadImagedatapost', methods=['POST'])
	def UploadImagedatapost():
	req_data = request.get_json()
	print (request.is_json)
	content = request.get_json()
	print (content)
	searchimagedata = req_data["imagedata"]
	print(searchimagedata)
	data = base64.b64decode(searchimagedata)
	fh = open("E:\\MobileTestImage2.jpg", "wb")
	fh.write(data)
	fh.close()
	return "data is " + searchimagedata
	#should be the last line of code
	app.run(host= '0.0.0.0')

view raw pythonflaskapi.py hosted with ❤ by GitHub

The following links were useful
link1, link2, chromeextension

REST API Interview Questions (Beginner Level)

REST API Interview Questions (Advanced Level)

Happy Learning!!!