BERT QnA Example

 Some examples are very good to pick on ideas and customize as we need. BERT based QnA Example

	#Thanks to https://theaidigest.in/question-answering-using-transformers-and-bert/
	#Experimenting for few linkedin job descriptions
	#!pip install transformers
	from transformers import pipeline
	nlp_qa = pipeline('question-answering')
	print(nlp_qa(context='Overall 8+ years’ experience working on highly distributed and scalable enterprise applications .5+ years’ experience and in depth understanding of Data sciences, Big data, AI ML technologies and platforms Experience in projects involving large scale-multi dimensional databases, complex business infrastructure, and cross-functional teams. Successfully launched two to three ML projects. Experience building and delivering complex systems, preferably in eCommerce areas that leverage various machine learning algorithms or technologies. Hands on experience developing and implementing machine learning algorithms (classification, clustering, regression, random forest etc.), using relevant programming languages, and big data tools. Experience in evaluating and making decisions around the use of various algorithms and tools for a project. Extensive experience with advanced ML techniques (neural networks, deep learning, reinforcement learning, SVM, PCA, etc.). Experience with open-source technologies, ML libraries, and programming languages. Good understanding of various cloud platforms and services it provides in implementing Big Data and AI ML solutions. Good appreciation of CI/CD process, tools and able to guide the team in decision making related to CI/CD strategies. Experienced with Agile development management tool (JIRA, Git/Bitbucket, Bamboo) Experience with API and Microservices architecture-based systems development and operations Excellent Communication, Leadership and Interpersonal skills.',
	question='What data science skills are required ?'))
	print(nlp_qa(context='Overall 8+ years’ experience working on highly distributed and scalable enterprise applications .5+ years’ experience and in depth understanding of Data sciences, Big data, AI ML technologies and platforms Experience in projects involving large scale-multi dimensional databases, complex business infrastructure, and cross-functional teams. Successfully launched two to three ML projects. Experience building and delivering complex systems, preferably in eCommerce areas that leverage various machine learning algorithms or technologies. Hands on experience developing and implementing machine learning algorithms (classification, clustering, regression, random forest etc.), using relevant programming languages, and big data tools. Experience in evaluating and making decisions around the use of various algorithms and tools for a project. Extensive experience with advanced ML techniques (neural networks, deep learning, reinforcement learning, SVM, PCA, etc.). Experience with open-source technologies, ML libraries, and programming languages. Good understanding of various cloud platforms and services it provides in implementing Big Data and AI ML solutions. Good appreciation of CI/CD process, tools and able to guide the team in decision making related to CI/CD strategies. Experienced with Agile development management tool (JIRA, Git/Bitbucket, Bamboo) Experience with API and Microservices architecture-based systems development and operations Excellent Communication, Leadership and Interpersonal skills.',
	question='What cloud computing stack are required ?'))
	print(nlp_qa(context='Overall 8+ years’ experience working on highly distributed and scalable enterprise applications .5+ years’ experience and in depth understanding of Data sciences, Big data, AI ML technologies and platforms Experience in projects involving large scale-multi dimensional databases, complex business infrastructure, and cross-functional teams. Successfully launched two to three ML projects. Experience building and delivering complex systems, preferably in eCommerce areas that leverage various machine learning algorithms or technologies. Hands on experience developing and implementing machine learning algorithms (classification, clustering, regression, random forest etc.), using relevant programming languages, and big data tools. Experience in evaluating and making decisions around the use of various algorithms and tools for a project. Extensive experience with advanced ML techniques (neural networks, deep learning, reinforcement learning, SVM, PCA, etc.). Experience with open-source technologies, ML libraries, and programming languages. Good understanding of various cloud platforms and services it provides in implementing Big Data and AI ML solutions. Good appreciation of CI/CD process, tools and able to guide the team in decision making related to CI/CD strategies. Experienced with Agile development management tool (JIRA, Git/Bitbucket, Bamboo) Experience with API and Microservices architecture-based systems development and operations Excellent Communication, Leadership and Interpersonal skills.',
	question='What is level of experience ?'))

view raw QnA.py hosted with ❤ by GitHub

Text Clustering - Did a decent job to cluster based on JD Types - Cloud, Server, ML etc..

	#!pip install -U sentence-transformers
	#https://colab.research.google.com/drive/182BUqhmnIXBGdefxf7LaMGJnssJNVZoP?usp=sharing#scrollTo=ZYWRFiyhzU0g
	from sentence_transformers import SentenceTransformer
	from sklearn.cluster import KMeans
	embedder = SentenceTransformer('distilbert-base-nli-stsb-mean-tokens')
	corpus = [
	"Advanced knowledge of SQL, R and / Or Python (4 years) with methodologies used for advanced analytics is required. Strong data visualization ability and ability to comprehend advanced Excel outputs to suggest alternate visual representation for concepts. Knowledge of statistics, optimization or related field. Experience with common data science toolkits - Pandas, Qlik Sense, Shiny, Plotly for better visualization",
	"Minimum 8+ years of experience in developing and delivering end to end applications using Java/J2EE, preferably in a Fintech company. Hands-on experience in designing and developing backend components, frontend components. Strong understanding of agile software practices. Strong understanding of application security standards and best practices. Prior experience in managing a team of engineers. Strong ability to collaborate with cross-functional teams including architects, engineers, quality engineering and operations teams to build solutions.Responsible for architecture assessments, cloud migration strategies and execution, legacy enterprise modernization and enablement across custom technologies and cloud migration opportunities",
	"Proven skills & experience in solution design and architecture, cloud-based solution using different cloud service provider and cloud native services, including designing the cloud infrastructure, designing the cloud application architecture, and designing the cloud security architecture. Ensuring technical viability and successful deployments, while orchestrating key resources and infusing key Infrastructure technologies (e.g. Windows and Linux IaaS, Security, Networking, etc.), and Application Development and DevOps technologies (e.g. App Service, containers, serverless, cloud native, etc.) as appropriateAdvanced problem-solving skills using programming concepts and Data Structures along with basic computer science fundamentals.",
	"Ability to extract meaningful scores from text and prior experience in text mining projects will be preferred. Prior experience in building models (data cleaning, dependent variable selection, independent variable study and understanding, variable reduction, bivariate analysis, variables grouping, logistic/linear model build, model validation, etc.) will be preferred. Actuaries/FRM/CFA/CQF/PRM certification would be a plus"
	]
	corpus_embeddings = embedder.encode(corpus)
	# Then, we perform k-means clustering using sklearn:
	from sklearn.cluster import KMeans
	num_clusters = 3
	clustering_model = KMeans(n_clusters=num_clusters)
	clustering_model.fit(corpus_embeddings)
	cluster_assignment = clustering_model.labels_
	cluster_assignment
	clustered_sentences = [[] for i in range(num_clusters)]
	for sentence_id, cluster_id in enumerate(cluster_assignment):
	clustered_sentences[cluster_id].append(corpus[sentence_id])
	for i, cluster in enumerate(clustered_sentences):
	print("Cluster ", i+1)
	print(cluster)
	print("")
	#Java and Cloud
	#ML
	#Finance
	#Not bad :)
	#Cluster 1
	#['Minimum 8+ years of experience in developing and delivering end to end applications using Java/J2EE, preferably in a Fintech company. Hands-on experience in designing and developing backend components, frontend components. Strong understanding of agile software practices. Strong understanding of application security standards and best practices. Prior experience in managing a team of engineers. Strong ability to collaborate with cross-functional teams including architects, engineers, quality engineering and operations teams to build solutions.Responsible for architecture assessments, cloud migration strategies and execution, legacy enterprise modernization and enablement across custom technologies and cloud migration opportunities', 'Proven skills & experience in solution design and architecture, cloud-based solution using different cloud service provider and cloud native services, including designing the cloud infrastructure, designing the cloud application architecture, and designing the cloud security architecture. Ensuring technical viability and successful deployments, while orchestrating key resources and infusing key Infrastructure technologies (e.g. Windows and Linux IaaS, Security, Networking, etc.), and Application Development and DevOps technologies (e.g. App Service, containers, serverless, cloud native, etc.) as appropriateAdvanced problem-solving skills using programming concepts and Data Structures along with basic computer science fundamentals.']
	#Cluster 2
	#['Advanced knowledge of SQL, R and / Or Python (4 years) with methodologies used for advanced analytics is required. Strong data visualization ability and ability to comprehend advanced Excel outputs to suggest alternate visual representation for concepts. Knowledge of statistics, optimization or related field. Experience with common data science toolkits - Pandas, Qlik Sense, Shiny, Plotly for better visualization']
	#Cluster 3
	#['Ability to extract meaningful scores from text and prior experience in text mining projects will be preferred. Prior experience in building models (data cleaning, dependent variable selection, independent variable study and understanding, variable reduction, bivariate analysis, variables grouping, logistic/linear model build, model validation, etc.) will be preferred. Actuaries/FRM/CFA/CQF/PRM certification would be a plus']

view raw TextCluster.py hosted with ❤ by GitHub

Unsupervised NER using BERT

Document search with fragment embeddings

Finbert

Finbert

Keep Exploring!!!