text: Data Science Stack Exchange is a question and answer site for Data science professionals, Machine Learning specialists, and those interested in learning more about the field. It is tedious to prepare the input and output pairs given the time series data. Stock market data space is highly dimensional and, as such, algorithms that try to exploit potential patterns or structure in the price formation can suffer from the so-called “curse of dimensionality”. Yes, we need to take into account the temporal properties of the data. It is provided by Patrick David and hosted on Kaggle. LSTM Autoencoders can learn a compressed representation of sequence data and have been used on video, text, audio, and time series sequence data. Jagadeesh23, October 29, 2020 . Can you apply the model to your dataset? Time series analysis has a variety of applications. The proposed model in this paper consists of three parts: wavelet transforms (WT), stacked autoencoders (SAEs) and long-short term memory (LSTM). Why is predicted rainfall by LSTM coming negative for some data points? Think of RNN as a for loop over time step so the state is kept. Finally, the TimeDistributed layer creates a vector with a length of the number of outputs from the previous layer. This book will guide you on your journey to deeper Machine Learning understanding by developing algorithms in Python from scratch! Multivariate LSTM Forecast Model Run the complete notebook in your browser. How do I provide exposition on a magic system when no character has an objective or complete understanding of it? Additionally, in almost all contexts where the term "autoencoder" is used, the compression and decompression functions are implemented with neural networks. Future stock price prediction is probably the best example of such an application. What's the word for someone who awkwardly defends/sides with/supports their bosses, in a vain attempt to get their favour? Think of your ReactJs, Vue, or Angular app enhanced with the power of Machine Learning models. The Overflow Blog How to write an effective developer resume: Advice from a hiring manager. Do the benefits of the Slasher Feat work against swarms? This guide will show you how to build an Anomaly Detection model for Time Series data. The skills taught in this book will lay the foundation for you to advance your journey to Machine Learning Mastery! Yet, the produced clusters visually separate the classes of ECG's. Not quite. Time-series forecasting with deep learning & LSTM autoencoders. Specifically, we will be designing and training an LSTM autoencoder using the Keras API with Tensorflow 2 as the backend to detect anomalies (sudden price changes) in the S&P 500 index. Autoencoders Neural Networks try to learn data representation of its input. EDIT 3 December 2018, I receive many questions over email. Let’s download and load into a Data Frame: Let’s have a look at the daily close price: That trend (last 8 or so years) looks really juicy. To subscribe to this RSS feed, copy and paste this URL into your RSS reader. In a sense, we’re forcing the model to learn the most important features of the data using as few parameters as possible. What is an auto encoder? SAEs is the main part of the model and is used to learn the deep features of financial time series in an unsupervised manner. Convolutional Autoencoders in Python with Keras. Is this kitchen wall parallel with the joists load-bearing? Making statements based on opinion; back them up with references or personal experience. We’ll use the model to find anomalies in S&P 500 daily closing prices. You can try to tune the model and/or the threshold to get even better results. This script demonstrates how you can use a reconstruction convolutional autoencoder model to detect anomalies in timeseries data. This book brings the fundamentals of Machine Learning to you, using tools and techniques used to solve real-world problems in Computer Vision, Natural Language Processing, and Time Series analysis. What is a "Major Component Failure" referred to in news reports about the unsuccessful Space Launch System core stage test firing? Anomaly here to detect that, actual results differ from predicted results in price prediction. In this project, we’ll build a model for Anomaly Detection in Time Series data using Deep Learning in Keras with Python code. It is one of the most commonly followed equity indices, and many consider it to be one of the best representations of the U.S. stock market. A lot of supervised and unsupervised approaches to anomaly detection has been proposed. You just combined two powerful concepts in Deep Learning - LSTMs and Autoencoders. Our Autoencoder should take a sequence as input and outputs a sequence of the same shape. We will use an LSTM Autoencoder Neural Network to detect/predict anomalies (sudden price changes) in the S&P 500 index. This article will see how to create a stacked sequence to sequence the LSTM model for time series forecasting in Keras/ TF 2.0. You might want to board the train. This article was published as a part of the Data Science Blogathon. Furthermore, some regular advices would be to standardize the input, change the activation functions (tanh worked well for me in the output layer) as well as the number of neurons per layer and the amount of layers in general. Timeseries anomaly detection using an Autoencoder. How early can you “catch” sudden changes/anomalies? For the CAE it looks reasonable but the other models lack some layers, or? We’ll use 95% of the data and train our model on it: Next, we’ll rescale the data using the training data and apply the same transformation to the test data: Finally, we’ll split the data into subsequences. The result is a model that can find anomalies in S&P 500 closing price data. We define the reconstruction LSTM Autoencoder architecture that expects input sequences with 30 time steps and one feature and outputs a sequence with 30 time steps and one feature. Basic Data Preparation 3. Use MathJax to format equations. Developers Corner. Specifically, we’ll be designing and training an LSTM Autoencoder using Keras API, and Tensorflow2 as back-end. Now we will split the time series data into subsequences and create a sequence of 30 days of historical data. Your first LSTM Autoencoder is ready for training. Eaga Trust - Information for Cash - Scam? -Wikipedia. The model doesn't usethe labels during training. How can autoencoders be used for anomaly detection of time time series data? To utilize the temporal patterns, LSTM Autoencoders is used to build a rare event classifier for a multivariate time-series process. To learn more, see our tips on writing great answers. you must be familiar with Deep Learning which is a sub-field of Machine Learning. It is an artificial neural network used for unsupervised learning of efficient codings. How can a GM subtly guide characters into making campaign-specific character choices? As we are aware that, real-life data is streaming, time-series data etc., where anomalies give… We will also create interactive charts and plots using Plotly Python and … Setup. About Keras Getting started Developer guides Keras API reference Code examples Computer Vision Natural language processing Structured Data Timeseries Audio Data Generative Deep Learning Reinforcement learning Quick Keras recipes Why choose Keras? Details about the data preprocessing steps for LSTM model are discussed. Our data is the daily closing prices for the S&P 500 index from 1986 to 2018. In this hands-on introduction to anomaly detection in time series data with Keras, you and I will build an anomaly detection model using deep learning. Community & governance Contributing to Keras » Code examples / Timeseries Timeseries. Learn how to solve real-world problems with Deep Learning models (NLP, Computer Vision, and Time Series). How can I achieve better compression? Build Machine Learning models (especially Deep Neural Networks) that you can easily integrate with existing or new web apps. Summary. The red dots (anomalies) are covering most of the points with abrupt changes to the closing price. Go from prototyping to deployment with PyTorch and Python! By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy. I compiled the most common questions into a FAQ at the end of this readme This repo presents a simple auto encoder for time series. What does children mean in “Familiarity breeds contempt - and children.“? Introduction. Overview. For time series data, recurrent autoencoder are especially useful. The data contains only two columns/features - the date and the closing price. I am familiar with using autoencoders to detect Fraud in credit card transactions, But my data is a time series one. Public Score . RepeatVector () repeats the inputs 30 times. A simple LSTM Autoencoder model is trained and used for classification. LSTM Autoencoder models have been successfully proposed for … Architecturally, the simplest form of an auto-encoder is a feedforward, non-recurrent neural net which is very similar to the multilayer perceptron (MLP), with an input layer, an output layer and one or more hidden layers connecting them. 8 min read. Recurrent Autoencoder. 1.12361. Adding return_sequences=True in LSTM layer makes it return the sequence. MathJax reference. Build LSTM Autoencoder Neural Net for anomaly detection using Keras and TensorFlow 2. Luckily, LSTMs can help us with that. How can a monster infested dungeon keep out hazardous gases? What are autoencoders? I take the ouput of the 2dn and repeat it “seq_len” times when is passed to the decoder. You will be able to design and build an LSTM autoencoder in Keras to detect anomalies in time series data. Some improvement in the accuracy over a Dense Autoencoder is found. Submitted by DimitreOliveira 2 months ago. The average loss for simple autoencoder is 14.28%, for convolutional autoencoder is 8.04%, for LSTM-autoencoder is 9.25%. Browse other questions tagged machine-learning time-series autoencoder or ask your own question. We would expect the autoencoder to do a really good job at reconstructing the digit, as that is exactly what the autoencoder was trained to do — and if we were to look at the MSE between the input image and the reconstructed image, we would find that it’s quite low. What results did you get? Its goal is to induce a representation (encoding) for a set of data by learning an approximation of the identity function of this data Id:X→X. The differences between au… Anomaly detection refers to the task of finding/identifying rare events/data points. What is the simplest proof that the density of primes goes to zero? Perhaps i should pay attention to other methods? site design / logo © 2021 Stack Exchange Inc; user contributions licensed under cc by-sa. The only difference is that the encoder and decoder are replaced by RNNs such as LSTMs. Did you vary the topology? The decoder ends with linear layer and relu activation ( samples are normalized [0-1]) The model has 2 layers of GRU. Asking for help, clarification, or responding to other answers. Autoencoder MLP with LSTM encoded feature Comparing models. As you can see, the denoised samples are not entirely noise-free, but it’s a lot better. The 1st is bidirectional. You can play around with the threshold and try to get even better results. Who must be present on President Inauguration Day? Here are the results: Still, we need to detect anomalies. You’ll learn how to use LSTMs and Autoencoders in Keras and TensorFlow 2. Let’s start with calculating the Mean Absolute Error (MAE) on the training data: We’ll pick a threshold of 0.65, as not much of the loss is larger than that. It visualizes the embeddings using both PCA and tSNE. How to use Keras TimeseriesGenerator for time series data; Home; About Me; Blog; Support; Posted by: Chengwei 2 years, 3 months ago You might have dealt with a predictive model whose task is to predict a future value based on historical data. Timeseries anomaly detection using an Autoencoder… Chosen by, gdown --id 10vdMg_RazoIatwrT7azKFX4P02OebU76 --output spx.csv, # reshape to [samples, time_steps, n_features], Step-by-step understanding LSTM Autoencoder layers, Train an Autoencoder on normal data (no anomalies), Take a new data point and try to reconstruct it using the Autoencoder, If the error (reconstruction error) for the new data point is above some threshold, we label the example as an anomaly. I show this on a dataset of 5000 ECG's. How to develop LSTM Autoencoder models in Python using the Keras deep learning library. Here are the basic steps to Anomaly Detection using an Autoencoder: Good, but is this useful for Time Series Data? Despite from that, AEs are thoroughly used for time series, especially LSTM+AE. View in Colab • GitHub source. The S&P 500, or just the S&P, is a stock market index that measures the stock performance of 500 large companies listed on stock exchanges in the United States. Learn why and when Machine learning is the right tool for the job and how to improve low performing models! This tutorial is divided into 3 parts; they are: 1. Podcast 290: This computer science degree is brought to you by Big Tech. Using the Autoencoder Model to Find Anomalous Data After autoencoder model has been trained, the idea is to find data items that are difficult to correctly predict, or equivalently, difficult to reconstruct. Here’s the little helper function for that: We’ll create sequences with 30 days worth of historical data: The shape of the data looks correct. When should you buy or sell? Above all, you should take care of the time series. TL;DR Detect anomalies in S&P 500 daily closing price. How can we make LSTM Autoencoder in Keras? Where is the antenna in this remote control board? It is not an autoencoder variant, but rather a traditional autoencoder stacked with convolution layers: you basically replace fully connected layers by convolutional layers. Here are the models I tried. The 2nd is not. Abstract. Hi to all, Issue: I’m trying to implement a working GRU Autoencoder (AE) for biosignal time series from Keras to PyTorch without succes.. How to optimally train deep learning model using output as new input. Why do small patches of snow remain on the ground many days or weeks after all the other snow has melted? Autoencoders are a type of self-supervised learning model that can learn a compressed representation of input data. It only takes a minute to sign up. People repeatedly ask me how to find patterns in time series using ML. Stack Exchange network consists of 176 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers. Build LSTM Autoencoder Neural Net for anomaly detection using Keras and TensorFlow 2. In the first part of this tutorial, we’ll discuss what autoencoders are, including how convolutional autoencoders can be applied to image data. Multivariate Multi-step Time Series Forecasting using Stacked LSTM sequence to sequence Autoencoder in Tensorflow 2.0 / Keras. What is the highest road in the world that is accessible by conventional vehicles? Download our Mobile App. Thanks See Also. The usual wavelet transforms and other features f… Figure 3: Reconstructing a digit from MNIST with autoencoders, Keras, TensorFlow, and deep learning. Since your input data consists of images, it is a good idea to use a convolutional autoencoder. Why such a big difference in number between training error and validation error? Clearly, the autoencoder has learnt to remove much of the noise. Some nice results! Article Videos. Input (1) Output Execution Info Log Comments (19) Best Submission. It can be unrolled into a feedforward network. The encoding should allow for output similar to the original input. So the input of the Autoencoder is the same as the output? This guide will show you how to build an Anomaly Detection model for Time Series data. Usually, we want to learn an efficient encoding that uses fewer parameters/memory. I got such results. Join the weekly newsletter on Data Science, Deep Learning and Machine Learning in your inbox, curated by me! One such application is the prediction of the future value of an item based on its past values. You’ll learn how to use LSTMs and Autoencoders in Keras and TensorFlow 2. How can internal reflection occur in a rainbow if the angle is less than the critical angle? rev 2021.1.18.38333, The best answers are voted up and rise to the top, Data Science Stack Exchange works best with JavaScript enabled, Start here for a quick overview of the site, Detailed answers to any questions you might have, Discuss the workings and policies of this site, Learn more about Stack Overflow the company, Learn more about hiring developers or posting ads with us, Autoencoders for the compression of time series, Tips and tricks for designing time-series variational autoencoders. Could you provide the head() of the input data? Some applications include - bank fraud detection, tumor detection in medical imaging, and errors in written text. Data preprocessing Time-series processing Regular LSTM model. My question is: is it practical to compress time series with losses using a neural network if the compression time does not matter? The RepeatVector layer simply repeats the input n times. Some of the approaches include - One-class SVMs, Bayesian Networks, Cluster analysis, and (of course) Neural Networks. In this blog post, we created a denoising / noise removal autoencoder with Keras, specifically focused on signal processing. "Autoencoding" is a data compression algorithm where the compression and decompression functions are 1) data-specific, 2) lossy, and 3) learned automatically from examples rather than engineered by a human. What was the DRAM refresh interval on early microcomputers? Here’s how to build such a simple model in Keras: There are a couple of things that might be new to you in this model. I am trying to use autoencoder (simple, convolutional, LSTM) to compress time series. Autoencoders with Keras, TensorFlow, and Deep Learning. Set return_sequences=True, … When the error is larger than that, we’ll declare that example an anomaly: Let’s calculate the MAE on the test data: We’ll build a DataFrame containing the loss and the anomalies (values above the threshold): Looks like we’re thresholding extreme values quite well. Time series prediction (forecasting) has experienced dramatic improvements in predictive accuracy as a result of the data science machine learning and deep learning evolution. As these ML/DL tools have evolved, businesses and financial institutions are now able to forecast better by applying these new technologies to solve old problems. Beginners Guide to Pyjanitor – A Python Tool for Data Cleaning . Successful. This Notebook has been released under the … Air Pollution Forecasting 2. We’ll use the model to find anomalies in S&P 500 daily closing prices. I am trying to use autoencoder (simple, convolutional, LSTM) to compress time series. Let’s create a DataFrame using only those: Finally, let’s look at the anomalies found in the testing data: You should have a thorough look at the chart. An AE expects to fit X on X, maybe you missed that? Time series analysis refers to the analysis of change in the trend of the data over a period of time. Thanks for contributing an answer to Data Science Stack Exchange! 24.11.2019 — Deep Learning, Keras, TensorFlow, Time Series, Python — 3 min read. Author: pavithrasv Date created: 2020/05/31 Last modified: 2020/05/31 Description: Detect anomalies in a timeseries using an Autoencoder. Training the model is no different from a regular LSTM model: We’ve trained our model for 10 epochs with less than 8k examples. To check for compression loss, I use the SMAPE formula. Since we focus on extracting the features from time series sensor data, we apply stacked LSTM Autoencoders to generate features, Mehdiyev, Lahann, Emrich, Enke, Fettke, and Loos / Procedia Computer Science 00 (2017) 000â€“000 which can be later feed to our classification algorithm, as described in the next sub-section. Some layers, or Angular app enhanced with the power of Machine Learning in your inbox, curated by!! Cae it looks reasonable but the other models lack some layers, or Angular app enhanced the! Prediction is probably the Best example of such an application with a length of the approaches -! A Dense Autoencoder is the same as the output validation error that the density primes! Learn data representation of input data set return_sequences=True, … you will be able design. ; they are: 1 combined two powerful concepts in Deep Learning Keras. Sudden price changes ) in the trend of the data contains only two columns/features the... Good idea to use LSTMs and autoencoders in Keras and TensorFlow 2 of ECG 's and closing! The other models lack some layers, or their favour Code examples / timeseries.! And autoencoders policy and cookie policy a simple LSTM Autoencoder Neural Net for anomaly detection model time. News reports autoencoder keras time series the data combined two powerful concepts in Deep Learning copy. Lay the foundation for you to advance your journey to Machine Learning series in an unsupervised.... Encoding that uses fewer parameters/memory of efficient codings of images, it is provided by Patrick David and on! Be designing and training an LSTM Autoencoder Neural Net for anomaly detection model for time series ) properties of approaches. Reasonable but the other models lack some layers, or responding to other answers sequence to sequence LSTM! To deeper Machine Learning in your inbox, curated by me price changes ) in the S & P daily. Anomalies ) are covering most of the input n times or Angular app with. Pavithrasv Date created: 2020/05/31 Last modified: 2020/05/31 Description: detect anomalies in time series in... Reasonable but the other snow has melted used for anomaly detection model for time series?... Governance Contributing to Keras » Code examples / timeseries timeseries bank Fraud,... Conventional vehicles autoencoders to detect anomalies in a vain attempt to autoencoder keras time series their favour for data Cleaning is... I use the model and/or the threshold and try to get even better results, Bayesian Networks Cluster. An AE expects to fit X on X, maybe you missed that more see! Digit from MNIST with autoencoders, Keras, TensorFlow, and ( of ). Breeds contempt - and children. “ model are discussed to the analysis of change in the accuracy over Dense... 14.28 %, for convolutional Autoencoder site design / logo © 2021 Exchange! Effective developer resume: Advice from a hiring manager, actual results differ from predicted results price... Time does not matter from that, actual results differ from predicted in... To get their favour ) to compress time series with losses using a Neural network to detect/predict anomalies sudden... Detection of time understanding of it: Still, we created a denoising / noise removal Autoencoder with Keras specifically. All the other snow has melted to find patterns in time series with losses a... Its past values subsequences and create a Stacked sequence to sequence Autoencoder in Keras to detect in. Tagged machine-learning time-series Autoencoder or ask your own question the LSTM model for time series data solve problems! Models in Python from scratch but my data is the main part of the data only... Designing and training an LSTM Autoencoder in TensorFlow 2.0 / Keras complete understanding of it for anomaly. Learning understanding by developing algorithms in Python from scratch snow remain on ground! That is accessible by conventional vehicles series, Python — 3 min read - bank Fraud detection, detection! Fewer parameters/memory convolutional, LSTM ) to compress time series data looks reasonable the! Data Cleaning with PyTorch and Python unsuccessful Space Launch system core stage test firing time-series Autoencoder ask! Why do small patches of snow remain on the ground many days or after. Abrupt changes to the task of finding/identifying rare events/data points monster infested dungeon keep out hazardous gases autoencoders a! Of supervised and unsupervised approaches to anomaly detection using Keras and TensorFlow 2 the weekly newsletter on Science. Threshold to get their favour learn data representation of input data to subscribe this... Objective or complete understanding of it around with the joists load-bearing series one in written text time time with... Enhanced with the joists load-bearing paste this URL into your RSS reader columns/features - the Date and the price! That you can easily integrate with existing or new web apps ; user contributions licensed under cc by-sa consists... Length of the time series Forecasting using Stacked LSTM sequence to sequence Autoencoder in TensorFlow 2.0 / Keras,. Detect anomalies in S & P 500 index Stack Exchange are replaced by RNNs such as LSTMs 3 ;... Of 30 days of historical data work against swarms are not entirely noise-free, but my data a. Hosted on Kaggle the points with abrupt changes to the analysis of change in the trend of approaches. Self-Supervised Learning model that can learn a compressed representation of its input two powerful concepts Deep... Questions over email detection refers to the decoder both PCA and tSNE compressed... Input data of primes goes to zero Date and the closing price.. X, maybe you missed that is that the density of primes to... ( NLP, computer Vision, and Deep Learning model using output as new input used for unsupervised Learning efficient... … 8 min read the approaches include - bank Fraud detection, tumor in... A Stacked sequence to sequence the LSTM model for time series using ML, actual results differ from results! Cookie policy but my data is a model that can find anomalies in S & P 500 daily prices! And validation error inbox, curated by me bosses, in a timeseries using an Autoencoder stock price prediction probably... Use the model and is used to learn the Deep features of financial time series data recurrent. The critical angle Learning in your inbox, curated by me S & 500! 3: Reconstructing a digit from MNIST with autoencoders, Keras, specifically on. Of input data consists of images, it is provided by Patrick and... Now we will split the time series Forecasting in Keras/ TF 2.0 despite from,! Using output as new input 2021 Stack Exchange seq_len ” times when is passed to the closing price actual. Is that the encoder and decoder are replaced by RNNs such as.., curated by me show you how to find anomalies in a vain to! Accessible by conventional vehicles Bayesian Networks, Cluster analysis, and Deep Learning library some layers or... Our tips on writing great answers this RSS feed, copy and paste this URL into your RSS.. The approaches include - bank Fraud detection, tumor detection in medical imaging, and Deep Learning, Keras TensorFlow! And when Machine Learning models ( especially Deep Neural Networks try to more... Data is autoencoder keras time series time series one algorithms in Python using the Keras Deep Learning which a. Questions over email remain on the ground many days or weeks after all the other snow autoencoder keras time series melted news! Science Blogathon Overflow blog how to improve low performing models series Forecasting in Keras/ TF 2.0 curated by!. To the analysis of change in the accuracy over a Dense Autoencoder is 14.28 %, for convolutional is... Timedistributed layer autoencoder keras time series a vector with a length of the data preprocessing steps for model. Using the Keras Deep Learning - LSTMs and autoencoders am familiar with Deep Learning advance journey! I receive many questions over email to tune the model and is to. To develop LSTM Autoencoder models in Python using the Keras Deep Learning models autoencoder keras time series especially Deep Networks. Are replaced by RNNs such as LSTMs two columns/features - the Date and the closing.... And outputs a sequence of the input and output pairs given the time series using! Points with abrupt changes to the task of finding/identifying rare events/data points will be to! For some data points 30 days of historical data 2021 Stack Exchange Inc ; user contributions licensed under by-sa. Difference in number between training error and validation error a denoising / noise removal Autoencoder with Keras,,! Has melted, Deep Learning models ( NLP, computer Vision, and Deep Learning, Keras,,! Dram refresh interval on early microcomputers head ( ) of the data convolutional, LSTM ) to compress time using... 3: Reconstructing a digit from MNIST with autoencoders, Keras,,... Is used to learn the Deep features of financial time series data, recurrent Autoencoder are especially.! With using autoencoders to detect anomalies ECG 's only difference is that the density of primes goes zero! Do small patches of snow remain on the ground many days or weeks after all the other snow has?... Our data is a sub-field of Machine Learning is the antenna in this control! A digit autoencoder keras time series MNIST with autoencoders, Keras, TensorFlow, and of! Data points, Deep Learning which is a  Major Component Failure '' referred to in reports! On X, maybe you missed that character choices Deep Neural Networks proof the... Separate the classes of ECG 's article will see how to develop LSTM Autoencoder Neural Net for anomaly refers. ) output Execution Info Log Comments ( 19 ) Best Submission, maybe missed. Road in the trend of the input data learn an efficient encoding that uses fewer.. A reconstruction convolutional Autoencoder model to find patterns in time series data prediction of the model to find patterns time... Deep Neural Networks ) that you can use a reconstruction convolutional Autoencoder is 14.28 %, LSTM-autoencoder... Date created: 2020/05/31 Description: detect anomalies in time series in unsupervised...


text:
User johndoe - MathOverflow most recent 30 from http://mathoverflow.net 2013-06-19T20:44:47Z http://mathoverflow.net/feeds/user/25577 http://www.creativecommons.org/licenses/by-nc/2.5/rdf http://mathoverflow.net/questions/127841/giving-topx-y-an-appropriate-topology/127922#127922 Answer by johndoe for Giving $Top(X,Y)$ an appropriate topology johndoe 2013-04-18T06:11:24Z 2013-04-19T09:24:04Z
It might be of interest to the original poster to know that $Top(X,Y)$ endowed with the compact-open topology guarantees at least one direction in the implication. In other words, if $F_*\colon X\times\mathbb{I}\to Y$ is continuous then $F\colon \mathbb{I}\to Top(X,Y)$ is. Hence you can safely interpret any homotopy as a path in the function space $Top(X,Y)$, but (allegedly) there are paths in $Top(X,Y)$ which do not correspond to homotopies.

I, for my part, would like to see a counterexample for the opposite direction, since all the counterexamples I know of seem to use a space different than $\mathbb{I}$.

Reference: Dugundji, Topology, Chapter XII, Theorem 3.1.
 http://mathoverflow.net/questions/104198/classify-mathbbrn-bundles/104203#104203 Answer by johndoe for Classify $\mathbb{R}^n$-bundles johndoe 2012-08-07T14:33:22Z 2012-08-07T18:45:26Z
I assume that by $\mathbb{R}^n$-fiber bundles you mean differentiable bundles with fiber diffeomorphic to $\mathbb{R}^n$. So you are forgetting the linear structure of $\mathbb{R}^n$, keeping only the smooth structure. It is a fact that $\textit{Diff}(\mathbb{R}^n)$ deformation retracts to $\textit{GL}(n,\mathbb{R})$, hence yes, the classification of rank $n$ real vector bundles is essentially the same as that of $\mathbb{R}^n$-fiber bundles. I don't know whether this holds at the topological level already (that is, whether $\textit{Homeo}(\mathbb{R}^n)$ deformation retracts to $\textit{GL}(n,\mathbb{R})$).

EDIT: it seems that at the topological level the two beasts are different, at least for some $n$. The result is contained in a paper by William Browder entitled "Open and closed disc bundles", Ann. of Math. (2) 83 (1966), 218-230. It is freely available on JSTOR. As a consequence of the results in that paper, there exists some $n$ for which $\textit{Homeo}(\mathbb{R}^n)$ does not deformation retract onto $\textit{GL}(n,\mathbb{R})$. As far as I know, $n>2$.
 http://mathoverflow.net/questions/133865/does-the-gluing-procedure-in-robert-walds-book-general-relativity-yield-a-haus Comment by johndoe johndoe 2013-06-16T10:05:25Z 2013-06-16T10:05:25Z have you tried looking at the equivalent condition that the quotient map be open and $\sim$ be a closed subset of $\times^2\bigsqcup_{U\in\mathcal{U}}\tilde{M}_U$? http://mathoverflow.net/questions/133700/every-real-holomorphic-hamiltonian-vector-field-on-a-kahler-manifold-is-killing Comment by johndoe johndoe 2013-06-14T09:15:19Z 2013-06-14T09:15:19Z @Yemon Choi: yes. http://mathoverflow.net/questions/133724/existence-of-geodesics-in-continuous-metrics Comment by johndoe johndoe 2013-06-14T09:11:49Z 2013-06-14T09:11:49Z this paper seems relevant: arxiv.org/abs/1212.6962 http://mathoverflow.net/questions/130472/monotone-homotopy/130503#130503 Comment by johndoe johndoe 2013-05-13T17:48:12Z 2013-05-13T17:48:12Z I don't understand. How is "Directed Homotopy Theory" related to the OP question? Thanks for any clue. http://mathoverflow.net/questions/127841/giving-topx-y-an-appropriate-topology/128362#128362 Comment by johndoe johndoe 2013-04-23T12:51:53Z 2013-04-23T12:51:53Z I am interested in the reference to Lawvere's paper, thank you in advance. http://mathoverflow.net/questions/127841/giving-topx-y-an-appropriate-topology Comment by johndoe johndoe 2013-04-18T05:53:16Z 2013-04-18T05:53:16Z @Amr: it seems you misplaced some * in the edited version http://mathoverflow.net/questions/127841/giving-topx-y-an-appropriate-topology/127853#127853 Comment by johndoe johndoe 2013-04-17T17:08:14Z 2013-04-17T17:08:14Z doesn't Isbell consider the case where I (as in the OP question) might run over the whole category of spaces? In other words, it seems to me that the OP question is a special case of the problem considered in Isbell's survey and might very likely have an affirmative answer. Am I wrong? http://mathoverflow.net/questions/127841/giving-topx-y-an-appropriate-topology Comment by johndoe johndoe 2013-04-17T17:03:14Z 2013-04-17T17:03:14Z to Anton Fetisov: why is the case I rather useless? http://mathoverflow.net/questions/127841/giving-topx-y-an-appropriate-topology/127847#127847 Comment by johndoe johndoe 2013-04-17T14:10:12Z 2013-04-17T14:10:12Z shouldn't A be the interval I in order to prove the inexistence of such a topology? http://mathoverflow.net/questions/121620/why-does-gln-have-no-spinor-representations Comment by johndoe johndoe 2013-02-12T17:20:05Z 2013-02-12T17:20:05Z i believe the missing word here is faithful. take a look at lemma 5.23 in lawson-michelsohn. http://mathoverflow.net/questions/121168/group-of-diffeomorphisms-of-a-manifold Comment by johndoe johndoe 2013-02-08T10:50:13Z 2013-02-08T10:50:13Z you might want to take a look at banyaga's book "the structure of classical diffeomorphism groups", 1997. http://mathoverflow.net/questions/118955/real-symmetric-matrix-has-at-least-one-real-eigenvalue-an-elementary-algebraic Comment by johndoe johndoe 2013-01-15T10:16:08Z 2013-01-15T10:16:08Z this question seems a duplicate of this one mathoverflow.net/questions/118626/… (and the OP has actually contributed to it by answering) http://mathoverflow.net/questions/116888/on-john-von-neumann-and-quantum-mechanics/116981#116981 Comment by johndoe johndoe 2012-12-21T18:27:43Z 2012-12-21T18:27:43Z too much alcohol during festivities? http://mathoverflow.net/questions/108560/what-is-a-good-book-on-topological-groups Comment by johndoe johndoe 2012-10-01T17:23:58Z 2012-10-01T17:23:58Z You might want to take a look at Hewitt-Ross (two volumes). http://mathoverflow.net/questions/107163/a-function-that-is-defined-everywhere-but-has-unknown-values Comment by johndoe johndoe 2012-09-14T13:06:45Z 2012-09-14T13:06:45Z Consider the function that is 1 everywhere if at least one proposed example suits your tastes, and 0 everywhere if whichever suggestion comes you can always find some reason to reject it.

format: plain


text:
Browse Questions

# Let $a_n=\large\frac{10^n}{n !}$ for $n=1,2,3.......$ Then the greatest value of $n$ for which $a_n$ is the greatest is :

$\begin {array} {1 1} (1)\;11 & \quad (2)\;20 \\ (3)\;10 & \quad (4)\;8 \end {array}$

(3) 10


text:
### Home > A2C > Chapter 2 > Lesson 2.2.3 > Problem2-149

2-149.

Rewrite each equation below. Then solve your new equation. Be sure to check your solution using the original equation.

1. $\left(n + 4\right) + n\left(n + 2\right) + n = 0$

Distribute the n.

Combine like terms, and factor the equation.

Simplify by Zero Product Property, then subtract 2 from both sides.

$n + 4 + \left(n^{2} + 2n\right) + n = 0$

$n^{2} + 4n + 4 = 0$

$\left(n + 2\right) \left(n + 2\right) = 0$

$n + 2 = 0$

$n = −2$

1. $\frac { 4 } { x } = x + 3$

Multiply both sides by x. Then set the equation equalto 0 and use the ZPP.

$x = −4 \text{ or } x = 1$


text:
Double phase obstacle problems with multivalued convection and mixed boundary value conditions

@article{Zeng2022DoublePO,
title={Double phase obstacle problems with multivalued convection and mixed boundary value conditions},
author={Shengda Zeng and Vicenctiu D. Ruadulescu and Patrick Winkert},
journal={Discrete and Continuous Dynamical Systems - B},
year={2022}
}
• Published 29 June 2021
• Computer Science
• Discrete and Continuous Dynamical Systems - B
<p style='text-indent:20px;'>In this paper, we consider a mixed boundary value problem with a double phase partial differential operator, an obstacle effect and a multivalued reaction convection term. Under very general assumptions, an existence theorem for the mixed boundary value problem under consideration is proved by using a surjectivity theorem for multivalued pseudomonotone operators together with the approximation method of Moreau-Yosida. Then, we introduce a family of the approximating…

References

SHOWING 1-10 OF 58 REFERENCES
Convergence analysis for double phase obstacle problems with multivalued convection term
• Mathematics
• 2020
Abstract In the present paper, we introduce a family of the approximating problems corresponding to an elliptic obstacle problem with a double phase phenomena and a multivalued reaction convection
Double phase problems with variable growth and convection for the Baouendi–Grushin operator
• Mathematics
• 2020
In this paper we study a class of quasilinear elliptic equations with double phase energy and reaction term depending on the gradient. The main feature is that the associated functional is driven by
On variational problems and nonlinear elliptic equations with nonstandard growth conditions
We study elliptic problems where the boundedness condition is “separated” from the coercivity condition, which leads to the loss of uniqueness, regularity, and some other properties of solutions. We
Existence results for double phase implicit obstacle problems involving multivalued operators
• Mathematics
Calculus of Variations and Partial Differential Equations
• 2020
In this paper we study implicit obstacle problems driven by a nonhomogenous differential operator, called double phase operator, and a multivalued term which is described by Clarke’s generalized
Double phase problems with variable growth
• Mathematics
Nonlinear Analysis
• 2018
Regularity for general functionals with double phase
• Mathematics
• 2017
We prove sharp regularity results for a general class of functionals of the type \begin{aligned} w \mapsto \int F(x, w, Dw) \, dx, \end{aligned}w↦∫F(x,w,Dw)dx,featuring non-standard growth


text:
# Finitely additive measure on Cartesian square of countable set

Let $$\mu$$ be a probability measure on $$(\omega, 2^\omega, \mu)$$ measure space which is finitely additive and $$\mu(A)=0$$ for finite sets. We can define as usual $$\mu^2$$ on semiring $$\mathcal{G}=\{A\times B~|~A,B\in 2^\omega\}\subset2^{\omega\times\omega}$$: $$\mu^2(A\times B) = \mu(A)\cdot\mu(B)$$ and extend it to the ring $$\langle\mathcal{G}\rangle$$ of subsets of $$2^{\omega\times\omega}$$ generated by $$\mathcal{G}$$. Then we can do Peano-Jordan extension, i.e. define: $$\mathcal{U} = \{X\in 2^{\omega\times\omega}~|~\inf_{X\subset S\in\langle\mathcal{G}\rangle}\mu(S)=\sup_{X\supset S\in\langle\mathcal{G}\rangle}\mu(S)\}$$ The question is: what we can say about $$\mathcal{U}$$? How can we describe it in set-theoretic terms. Is it, for example, ring of subsets isomorphic to $$2^\omega$$ or it coincides with $$2^{\omega\times\omega}$$ (which is isomorphic to $$2^\omega$$) or something else? Are $$(\omega, 2^\omega, \mu)$$ and $$(\omega\times\omega,\mathcal{U},\mu^2)$$ isomorphic?

• It is not $2^{\omega\times\omega}$. Indeed, $\{(m,n)\mid m\leq n\}\notin\mathcal{U}$. For if $A\times B\subseteq \{(m,n)\mid m\leq n\}$, then $A$ must be finite and therefore $\mu^2(A\times B)=0$. A similar argument applies to the complement $\{(m,n)\mid m> n\}$. – Michael Greinecker Mar 18 at 13:28

Here is something that might be of interest to you, unfortunatley this is too long to be a comment.

As Michael Greinecker has said, $$\mathcal U\neq 2^{\omega\times\omega}$$ and thus $$(\omega, 2^\omega, \mu)$$ and $$(\omega\times\omega, \mathcal U, \mu^2)$$ cannot be isomorphic. However, in the case where $$\mu$$ comes from a non-principal ultrafilter on $$\omega$$ (i.e. $$\mu(A)\in\{0, 1\}$$ for all $$A\subseteq\omega$$), $$\mu^2$$ can always be extended to $$\mu^2\subseteq\lambda$$ so that in fact $$(\omega, 2^\omega, \mu)\cong(\omega\times\omega, 2^{\omega\times\omega}, \lambda)$$.

To see this, let $$\mathcal F$$ be the ultrafilter derived from $$\mu$$. It is straightforward to show that $$\mathcal U=\{X\subseteq\omega\times\omega\mid \exists A, B\in\mathcal F\ (A\times B\subseteq X\vee A\times B\subseteq \omega\setminus X)\}$$ and for $$X\in\mathcal U$$, $$\mu^2(X)=1$$ iff $$\exists A, B\in\mathcal F\ A\times B\subseteq X$$.

We now start to define the bijection $$f:\omega\times\omega\rightarrow\omega$$ that will induce the isomorphism in the end.

Claim 1 There is a partition $$(P_n)_{n\in\omega}$$ of $$\omega$$ into infinite $$\mu$$-nullsets so that $$\{n\in\omega\mid n\in P_n\}\in\mathcal F$$.

Proof: Let $$D$$ be any infinte, co-infinite set in $$\mathcal F$$ and choose $$(P_n)_{n\in\omega}$$ to be any partition of $$\omega$$ with $$P_n\cap D=\emptyset$$ for $$n\notin D$$ and $$n\in P_n$$ for $$n\in D$$. $$\square$$

Next we choose for each $$n$$ an enumeration $$P_n=\{m^n_k\mid k\in\omega\}$$ so that if $$n\in P_n$$ then $$n=m^n_n$$. Define $$f$$ as $$f(n, k)=m^n_k$$. This is a bijection and for $$\mathcal F$$-almost all $$n$$ we have $$f(n, n)=n$$. Now let $$\lambda(X)=\mu(f[X])$$ for $$X\subseteq\omega\times\omega$$ be the pullback of $$\mu$$ via $$f$$.

Claim 2 $$\lambda$$ extends $$\mu^2$$.

Proof: Assume $$X\in\mathcal U$$ and $$\mu^2(X)=1$$. Then there are $$A,B\in\mathcal F$$ with $$A\times B\subseteq X$$. Let $$D=\{n\in\omega\mid n\in P_n\}=\{n\in\omega\mid n=m^n_n\}\in\mathcal F$$. We have $$(D\cap A\cap B)^2\subseteq X$$ and thus $$f[X]\supseteq f[(D\cap A\cap B)^2]\supseteq D\cap A\cap B\in\mathcal F$$ which gives $$\lambda(X)=1$$. If otherwise $$\mu^2(X)=0$$ then the argument above shows $$\lambda(\omega\setminus X)=1$$, hence $$\lambda(X)=0$$. $$\square$$

This at least gives an embedding $$(\omega\times\omega, \mathcal U,\mu^2)\rightarrow (\omega\times\omega, 2^{\omega\times\omega}, \lambda)\cong (\omega, 2^\omega, \mu)$$.

• Thank you. I can add that in case of selective ultrafilter your embedding becomes isomorphism. Unfortunately in my case the measure is not ultrafilter. – ar.grig Mar 19 at 4:58


text:
# Gas station without pumps

## 2016 April 4

Filed under: Circuits course — gasstationwithoutpumps @ 22:44
Tags: , , ,

I spent all day Sunday grading the first set of lab reports.  I was expecting 24 reports of about 3 pages each, but I got 25 averaging about 5 pages each.  I think that the reports were a bit better this year than at corresponding times in previous years, but I did not get my grading done until almost midnight Sunday night, keeping me from getting much else done this weekend.

(I did manage to get my hair cut and to build a new strobe stand with room for 20 of my LED boards, which should give 1800 lumens during the flash. With a duty cycle of only 1/65, I don’t think that I need heat sinks on the boards for the strobe, as the average current should be only 40mA, though the peak current will be about 2.6A.)

In class on Monday, I gave students some group feedback on their writing, plus a couple of $\LaTeX$ pointers, then took questions, some of which were about writing, but most were about the optimization of the fixed resistor in the voltage divider for the resistance-to-voltage converter in the thermistor lab.  I showed them how to set that up, but did not try to solve it in class.

After class, when I was making up the key (redoing all the problems—I don’t like just looking up results—refreshing my memory on how to solve the problems by resolving them is best), I ran into a little trouble doing the optimization. I used to be able to just ask Wolfram Alpha to solve the differential equation, but their newer parser seems to be much harder to convince to do anything.  I eventually gave up and used a cruder tool to just take the second derivative and solved for the resistance by hand.  That was faster than the time I wasted trying to get Wolfram Alpha to do anything useful.  (I suspect that they have deliberately crippled it, to make people pay for Mathematica.)

Monday afternoon and evening (from about 1:30 to 7:45) was spent grading the first pre-lab homework.  Again the results are a little better than previous years, but there were 9 prelabs fewer than I expected (3 students have dropped already and 6 did not do the prelab).  I hope that those who did not do the prelab were just confused about when it was due, and not starting a trend towards coming to class and lab unprepared. I also hope that no more students drop—this class is not a weed-out class, though it is a lot of work.

Back in January, Mike wanted to know where I ended up doing my grading. Sunday I did my grading in my breakfast room, with the laptop on the floor where I could get to it if I really needed to look something up, but where it was not a constant temptation to goof off.  On Monday, I worked in my office on campus, where the e-mail was a minor distraction that I checked between problems.  (For the prelabs, I graded the entire stack for problem 1, then the entire stack for problem 2, and so forth.  This makes for more consistent and faster grading than grading a student at a time, but it would be faster still if the students didn’t put their answers in random order on what they turned in.) I’ll probably continue with weekend grading in the breakfast room and prelab grading in my office until the distractions get to be too much—then I’ll look for a coffeeshop to grade in.

## 2015 April 8

### Optimization and model fitting went well today

Filed under: Circuits course — gasstationwithoutpumps @ 21:47
Tags: , , , ,

Today’s lecture in BME 101 (the Applied Electronics for Bioengineers class) went very smoothly. I started with a little light entertainment: a video about blowing up capacitors, sent to me by Jameco: http://www.jameco.com/Jameco/content/incorrectly-using-capacitors.html This video reinforced the message that I had given them in lab last week about electrolytic capacitors.

I had two topics to cover: the optimization problem from last weekend’s homework, which only one student had managed to do, and model fitting to extract parameters for the thermistor model from the data collected in lab yesterday.

I gave the students the choice of order for the topics, and they wanted the optimization first.  The problem is fairly straightforward from an engineering standpoint: we want to maximize the sensitivity of the circuit that converts temperature to voltage at a specific operating temperature Topt. I first sketched very rough plots of what the voltage would look like as a function of temperature: monotone increasing but not particularly linear. I reminded them that sensitivity is the change in output that you get for a given change in input: dV/dT.  I coaxed out of the class the idea that to maximize the function dV/dT, we need to take its derivative and set it to 0: $\frac{d^2 V }{d T^2} = 0$.  We only have one free variable in our circuit to change (the load resistor R), so we need to take that equation and solve it for R, to get the value of R that maximizes sensitivity.

I reminded them of the simple model for the resistance of a thermistor that we’d had on Monday: $R_{T} = R_{\infty} e^{B/T}$, and they had no trouble coming up with the equation for the output voltage $V = V_{in} \frac{R}{R+R_{\infty} e^{B/T}}$.  I then suggested using Wolfram Alpha to solve the equation, and switched from the chalkboard to the screen to type

solve d^2 (R/(R+S e^(B/T)))/ dT^2 = 0 for R

(changing R to S, to avoid confusing Wolfram Alpha with subscripts).
Wolfram Alpha conveniently replied with two solutions: R=0 (which would result in a constant 0 output voltage, clearly a minimum sensitivity) and $R= S e^{B/T} \frac{B-2 T}{B+ 2T}$. I pointed out that the first part was just the resistance of the thermistor at the temperature we were optimizing for, and the second term scales that down a little.

We then moved on to fitting the model. I showed three data sets that students had sent me—one which was a little messy, but still quite usable, one that was a little better, and one that was really beautiful.  I continued with the really good data set.  I explained that gnuplot tries to minimize the error on the y-axis for a function of things on the x-axis, and the students decided that minimizing the temperature error for a given resistance was probably best here. (I suspect it doesn’t make much difference with this data, but it is good to get them to think about that as a decision that needs to be made for each modeling problem.)

On the board, I turned around the equation for the model, to get temperature in terms of R: $T = \frac{B}{\ln R - \ln R_{\infty}}$.

I then developed the gnuplot script for the problem live, debugging as I went.  Some of the errors to debug were ones I introduced deliberately, others were inadvertent, but all were good for teaching both the process of debugging and the notion of doing sanity checks.  My first mistake was an inadvertent one: I capitalized a variable differently in the parameter list and in the function body, and gnuplot is case-sensitive.  This was very easy to find and fix, and gave me an opportunity to tell students that gnuplot was sensitive to case, since that had not come up previously. My second mistake was a deliberate one: I typed in the model exactly as we had derived it on the board, knowing full well that the model was for °K, but the data was for °C.  I’d even reminded students of that earlier, when we were doing the optimization problem.

I then ran the fit in gnuplot and asked students if we were done, or if there was a sanity check we could do.  A couple of them asked if we could plot the model and the data on the same plot, so I did that. The fit was very obviously completely wrong. So I asked the students what was wrong and how we could fix it. I did dice-assisted cold calling to put 3 or 4 students on the spot, getting not very useful answers, then took answers from a couple of students who raised their hands. The second student pointed out the °C vs °K problem.

So I fixed the formula and ran the fit again, expecting everything to work fine.  It didn’t! So we went into debugging mode, trying to see if starting with better estimates of B and R would help. Nope. Then I realized that I had made a second inadvertent error: I’d put in 273.15 with the wrong sign! I fixed that, ran the fit again, and again things failed (producing negative values for R).  This time, though when I put in better initial estimates, everything converged to a very good fit, which gave me a teachable moment about the need to have decent estimates before fitting, so that the optimization algorithm that did the fitting could converge.

The final script was

temp(R,B,Rinf) = B / ( log(R) - log(Rinf)) - 273.15
B=3000; Rinf=1e-4
fit temp(x,B,Rinf) 'student-thermistor-data-3.txt' using 2:1 via Rinf,B
plot 'student-thermistor-data-3.txt' using 2:1, \
temp(x,3435,10*exp(-3435/(25+273.15))), \
temp(x,B,Rinf)

The second curve of the plot is using the data sheet values for the B value and for R25°C=10kΩ.
Here is the plot produced:

Notice that I did not have axis labels, title, nor echoing of the parameters in the key. We covered that last week and I’m hoping that students can carry skills over from one week to the next.

The model fit is slightly better than the data sheet values, but the data sheet values were closer than I had expected based on previous years’ fits. I think that it may be that the students had a properly calibrated thermometer (I’d removed any digital thermometers that reported ice water as warmer than 0.5°C—about 1/3 of them were that far out of calibration), and that this particular pair of students had been very careful with their measurements.


text:
# BGA soldering

A

#### Active8

Jan 1, 1970
0
hi:

i think win asked about using an oven. here's a snip from something that
turned up in a search:

******************
I recently came across and found the idea and the performances
interesting
as I'm balancing about using BGAs for homebrewing.
They have "selected" 3 or 4 toaster ovens and as far as I understand the
important thing is that they have the power resistors on the bottom side
of
the oven. This way the IR heat the PCB and then the solder past. This
ensure
a good temperature uniformity across all the PCB's surface (even under
the
component package) and a good temperature control.

To ensure an optimum soldering quality, i.e. follow as closely as
possible
the recommended temperature profile, you could one or several
thermocouple
on a dummy board and make some measures to optimize the process.

There's still the problem of solder past dispensing which is normally
done
by printing through a laser cut steal stencil. That's of course out of
question for personnal use (cost).
******************

note: in case anyone hasn't read the page on reflowing solder paste in
an oven with SMD parts, the paste was just laid down as a bead across
all the pads and the surface tension lined up the part and cleaned
everything up when the paste reflowed.

FWIW,

brs,
mike

T

#### Terry

Jan 1, 1970
0
Active8 threw some tea leaves on the floor
and this is what they wrote:

There's still the problem of solder past dispensing which is normally
done
by printing through a laser cut steal stencil. That's of course out of
question for personnal use (cost).

I can say from experience a that silk screened screen works almost as
well and have produced around 2000 smt boards a day using this method.

I've also used the laser cut stainless stencil method, both have their
pros and cons.

Don't be afraid to use a silk screen, it works. All you need do is save
up for the solder paste ;-)

A

#### Active8

Jan 1, 1970
0
Active8 threw some tea leaves on the floor
and this is what they wrote:

I can say from experience a that silk screened screen works almost as
well and have produced around 2000 smt boards a day using this method.

I've also used the laser cut stainless stencil method, both have their
pros and cons.

Don't be afraid to use a silk screen, it works. All you need do is save
up for the solder paste ;-)
i *know* you realize that stencil thing was a quote from someone else...

none the less, thanks for confirming my suspicions. IIRC in "Printed
Circuits Handbook" - Coombs goes to some length discussing the
fabrication of those steel stencils and how undercutting, etc. can foul
up the paste. who wants to screw around with steel and failure analysis
when a silk screen will do?

now since we have a flame war going elsewhere and this is a topic i've
been researching, perhaps you could share some of your experiences with
me. crap! i was hoping to find the latest file i found on making a
screen. it mentioned three methods and parts were unclear.

i'm really interested in knowing what works best for you and how to get
started. why don't we start with the solder paste screen while i try to
dig up that article on liguid imageable solder mask and etch resist
screening.

TIA,
mike

G

#### Garrett Mace

Jan 1, 1970
0
I can say from experience a that silk screened screen works almost as
well and have produced around 2000 smt boards a day using this method.

I've also used the laser cut stainless stencil method, both have their
pros and cons.

Don't be afraid to use a silk screen, it works. All you need do is save
up for the solder paste ;-)

I wonder if a silk screen for solder paste could be constructed with the
toner-transfer method. Print a positive and iron it into the material, then
use that to fill a negative on *another* piece of material.

Nah, stuff would probably move around too much. Maybe you could use a
positive to mask etch-resist onto a piece of thin metal shim stock, and then

A

#### Active8

Jan 1, 1970
0
On Sun, 12 Oct 2003 14:33:19 GMT, Active8, said...
note: in case anyone hasn't read the page on reflowing solder paste in
an oven with SMD parts, the paste was just laid down as a bead across
all the pads and the surface tension lined up the part and cleaned
everything up when the paste reflowed.
of course, with BGAs you don't need paste.

mike

A

#### Active8

Jan 1, 1970
0
On Sun, 12 Oct 2003 14:33:19 GMT, Active8, said...
note: in case anyone hasn't read the page on reflowing solder paste in
an oven with SMD parts, the paste was just laid down as a bead across
all the pads and the surface tension lined up the part and cleaned
everything up when the paste reflowed.
of course, with BGAs you don't need paste. te point is that the surface
tension of the solder *may* pull the chip in as it does for solder
paste. think of how you can pick up leads with an iron or get a chip cap
to follow the iron around if you don't hold it down.

mike

A

#### Active8

Jan 1, 1970
0
I wonder if a silk screen for solder paste could be constructed with the
toner-transfer method. Print a positive and iron it into the material, then
use that to fill a negative on *another* piece of material.

i really think the normal sesitized emulsion method would easier
Nah, stuff would probably move around too much. Maybe you could use a
positive to mask etch-resist onto a piece of thin metal shim stock, and then

check the links john gave in my toner xfer thread. they claim you can
etch metal (at least brass - don't ask me why brass is speacial) with
the film in the first link. i don't see why you couldn't do it with the
PnP blue or wet, though. the dynaart/pulsar site doesn't mention
anything special about their product that prevents undercutting of the
metal. IIRC Coombs doesn't mention using a trapezoidally (is that even a
word? ) undercut stencil, mirror imaged and flipped, so that it
doesn't lift the paste off the board. he may have mentioned that it
would be hard to squegee the paste in that way, though. i might play
with it one day, i think i know the trick. i also found another link
while checking out photoimagable solder mask. a company that etches
steel and aluminum stiffeners. Coates or someone. it's all doable.

but from what i've read, a silk screen will do just fine. i've seen
elargements of screened images and for paste and soldermask, it's fine.
i see resolutions that might work for etch-resist, also, but i'm
skeptical as to whether this would be acceptable. the enlarged images
i've seen were for traces and one of 'em looked, let's say "not too

let's see what Terry has to say about the pros and cons if he'll be so

crap! i can't find the link.

now here's the real ball-buster if 150 ain't too much... Ed (GPS) supplied this in this group. see: Re: Succes - first finepitch device soldered www.stencilsunlimited.com brs, mike H #### Hal Murray Jan 1, 1970 0 of course, with BGAs you don't need paste. I think they normally use paste to get the flux in there. A #### Active8 Jan 1, 1970 0 I think they normally use paste to get the flux in there. depends on what you mean by paste. the "balls" in "Ball Grid Array" are made of solder and you add flux. brs, mike G #### Georg Acher Jan 1, 1970 0 |> depends on what you mean by paste. |> |> the "balls" in "Ball Grid Array" are made of solder and you add flux. This is not true for all packages. There are a few BGA-packages made of ceramic with so much weight, that they would squeeze out the solder when warmed up. Thus, their balls don't melt (non-eutectic) and you need solder paste for proper contact. Normal packages (ie. the cheap epoxy-based ones) have solder balls and only need some flux. K #### Klaus Vestergaard Kragelund Jan 1, 1970 0 Terry said: Active8 threw some tea leaves on the floor and this is what they wrote: I can say from experience a that silk screened screen works almost as well and have produced around 2000 smt boards a day using this method. I've also used the laser cut stainless stencil method, both have their pros and cons. Don't be afraid to use a silk screen, it works. All you need do is save up for the solder paste ;-) -- I think I have missed that thread. Do you just use the silkscreen as the stencil for applying the paste? Could you direct me to the original thread, a link perhaps? Thanks Klaus D #### Da Man Jan 1, 1970 0 I think it was future active or digikey that sells little disposeable stick on stencils for BGA rework. A #### Active8 Jan 1, 1970 0 I think I have missed that thread. Do you just use the silkscreen as the stencil for applying the paste? Could you direct me to the original thread, a link perhaps? it was archived at google IIRC. could have been an archive or search hit of a web forum. but there was no mention of silk screen AFAIK. it was about pcb fab and the poster just mentioned the stainless stencil. you'll find more stuff on this than you can shake a tree at via google. don't worry, i think i got all the relevant stuff as far as toasters are concerned so you didn't miss much. i just found that snip in my note. IIRC, win hill or someone was asking about BGAs in this group and the question came up about reflowing them in toasters. brs, mike T #### Terry Jan 1, 1970 0 Active8 threw some tea leaves on the floor and said: i *know* you realize that stencil thing was a quote from someone else... Opps, your marks for clairvoyance aren't too high ;-) none the less, thanks for confirming my suspicions. IIRC in "Printed Circuits Handbook" - Coombs goes to some length discussing the fabrication of those steel stencils and how undercutting, etc. can foul up the paste. who wants to screw around with steel and failure analysis when a silk screen will do? I found the stainless stencil to be nice and accurate, but it had problems of paste leaking underneath. now since we have a flame war going elsewhere Eww, I'm not trying to start any flame wars and this is a topic i've been researching, perhaps you could share some of your experiences with me. crap! i was hoping to find the latest file i found on making a screen. it mentioned three methods and parts were unclear. I had my silk screens made at a local screen printer. I'd supply the artwork and they would make the screen. It all went together in a simple wooden jig, the pcb aligned by hand into two pcb thickness stops which were then fixed to the base of the jig. Apart from careful usage and cleaning the screen clean after each run they were fine for the task. Lets face it, IR reflow allows for a lot of errors anyway including pick and place machine alignment errors. i'm really interested in knowing what works best for you and how to get started. why don't we start with the solder paste screen while i try to dig up that article on liguid imageable solder mask and etch resist screening. My experience is getting a bit old now, and I imagine that there are many new techniques around. I used the standard solder paste that comes in a300 tube, it was
squeeged onto the silk screen carefully by a experienced operator
and then the parts were placed by an Amistar SMT P&P machine with a

From there it was into the IR reflow oven and out the other end to
inspection and testing.

I guess like many things, when you work with them everyday, they soon
seem very simple.

I haven't made any SMT assemblies (professionally) for years now, but
I'm thinking of setting up a small home system at the moment to compliment
my little home development workshop.

Pics of my setup and pcb efforts are at :-
http://milkstone.d2.net.au/software/pcb.html

T

#### Terry

Jan 1, 1970
0
Active8 threw some tea leaves on the floor
and said:
i really think the normal sesitized emulsion method would easier

I do as well, tho I've never made a silk screen myself.

Some kind of light sensitive emulsion would do, perhaps even PVC (the
whitish clear liquid) ?

A friend uses that as a photosensitive coating for etching aluminium
labels, and it works nicely judging from the results.

but from what i've read, a silk screen will do just fine. i've seen
elargements of screened images and for paste and soldermask, it's fine.
i see resolutions that might work for etch-resist, also, but i'm
skeptical as to whether this would be acceptable. the enlarged images
i've seen were for traces and one of 'em looked, let's say "not too

let's see what Terry has to say about the pros and cons if he'll be so

From my pov, and I'm not the expert here by any means:

Stencil:
Pros: Nice and accurate in terms of area to be covered and *depth* of
solderpaste.
Cons: Can get solder paste running underneath and smearing on the pcb.

Plus hard to make at home without a 50W NC laser ;-)

Silk screen:
Pros: Accurate enough, the solder paste flows around the silk strands.
easy to make compared to stencils, low in cost.
Cons: depth of solder paste is hard to control, needs an experienced op
to get repeatable results, must have good tension etc.

T

#### Terry

Jan 1, 1970
0
Klaus Vestergaard Kragelund threw some tea leaves on the floor
and said:
I think I have missed that thread.

Sorry Klaus I missed the whole thread as well, and just jumped in as is
my way
Do you just use the silkscreen as the
stencil for applying the paste?

I used to be the technical manager of a small SMT assembly plant. We
had one Amistar SMT P&P machine, a couple of Dynapert leaded p&p
machine, a die bonder and a wire bonder, plus re-flow oven and solder
bath etc.

We used silk screens to apply the solder paste at that time. Later in
another job, I came across stainless stencils and so I have had
experience with both methods.

The silk screen method is really simple. The screen has been coated
with a photosensitive paint and exposed thru the CAD generated artwork
which contains the images of all the SMT footprints (or the inverse).

So where a footprint exists on the pcb, the that area of the silk is
clear and will allow solderpaste to be squeegeed thru onto the pcb.

The operator makes a thin line of solderpaste at the top of the screen
after the pcb has been placed below into its alignment holders and one
pass of the squeegee is done. The screen is then lifted off (its hinged
at the top or side) and the pcb removed. The SMT parts are then placed,
naturally they stick to the solderpaste and even a decent knock against
a wall fails to move them!

If too little paste is placed, a second squeegee can be done, but this
isn't a great idea.

Oh, one more thing, the silk screen sits *above* the pcb by a few
millimeters and in theory only the area below the squeegee comes into
contact with the pcb, unlike a stencil which lays flat on the pcb.

T

#### Terry

Jan 1, 1970
0
Active8 threw some tea leaves on the floor
and said:
On Sun, 12 Oct 2003 14:33:19 GMT, Active8, said...

of course, with BGAs you don't need paste. te point is that the surface
tension of the solder *may* pull the chip in as it does for solder
paste. think of how you can pick up leads with an iron or get a chip cap
to follow the iron around if you don't hold it down.

mike

I found that this is true, however one needs a little more paste for
the reflow to move the SMT parts into place, and this makes the final
job a little less neat due to the extra solder and resin present.

With perfect alignment of the smt part, less solder paste can be used
and the board is ready to go without any cleanup, however too little
solder paste and some parts may "toombstone" or stand up slightly with
one end not touching the pcb.

This naturally is a "bad thing" (tm)

T

#### Terry

Jan 1, 1970
0
Active8 threw some tea leaves on the floor
and this is what they wrote:

<deletia>
I agree that the temperature of the "oven" should be well regulated
and follow the accepted profile of reflowing solderpaste.

Our IR commercial reflow oven was a bit like a pizza makers oven and
had about 5 heat plates above and below a metal mesh conveyor belt.

Each bank had its own temperature controller and the settings could be
altered, as could the speed of the conveyor belt.

The unit was about 10 feet long and the boards went thru in about 3
minutes (from memory).
note: in case anyone hasn't read the page on reflowing solder paste in
an oven with SMD parts, the paste was just laid down as a bead across
all the pads and the surface tension lined up the part and cleaned
everything up when the paste reflowed.

This makes me cringe because too much solder paste can be a real
problem and wicking *can* occur.

If that technique works, it's got to be more luck than anything else
imho.

I know that working with SMT is a hassle in many ways (especially at my
age and eyesight) but I don't think that trying to make it easy by just
placing solder paste across all the pins of a part is a process that
will be very reliable.

I imagine most of us here just make a few prototypes at a time?

In that case we need something that's as simple and cost effective as
possible, but what's the answer ?

I think that a syringe and a good eyeglass may allow the dispensing of
solder onto each pad of small prototype pcb with reasonable results ?

Or some easy way to produce a silk screen, but I haven't looked into
this yet.

Can some laser printer paper manufacturer please develop a semi stiff
silk A4 sheet please ? ;-)

Hmm, whats to stop someone using a a4 sticky backed paper sheet and
sticking some silk screen to it, then running thru a laser ?

If the silk melts, its *your problem*, but then again what about a fine

If the brass or the cut ends of the brass screen damage the laser toner
roller, once again its *your problem* not mine!

I think there are a few workable ideas here ?

I've got an expendable laser and nearly empty cart, so if I can find the
materials, I'll give it a go as soon as I can

A

#### Active8

Jan 1, 1970
0
Active8 threw some tea leaves on the floor

Opps, your marks for clairvoyance aren't too high ;-)

oops. it was clearly mentioned and delineated within asterisks ;-)
Eww, I'm not trying to start any flame wars

and i'd rather have a discussion with you than read that bs. it looks
like normal posting is back up. weekend's over and all. and my friggin'
(hmm... how many days has it been) or the damn think takes naps.
I had my silk screens made at a local screen printer. I'd supply the
artwork and they would make the screen.

ok, i'm planning on making my own for now. done a little research.

i noticed you mentioned PVC as a photosensitive emulsion. i didn't know
that it was used like that. anything else you can add to that. i know
nothing of PVC other than pipe and it's various uses.<snip>

brs,
mike

A

#### Active8

Jan 1, 1970
0
Active8 threw some tea leaves on the floor
and this is what they wrote:

<deletia>
I agree that the temperature of the "oven" should be well regulated
and follow the accepted profile of reflowing solderpaste.

profiles have been run on 4 toasters, but the pics can't be read and i'm
not convinced it's right because you have to tailor the profile for
certain situations.
Our IR commercial reflow oven was a bit like a pizza makers oven and
had about 5 heat plates above and below a metal mesh conveyor belt.

and those toasters tested had only top heaters which was claimed to be a
good thing. that's something else with which i don't neccessarily agree.
Each bank had its own temperature controller and the settings could be
altered, as could the speed of the conveyor belt.

The unit was about 10 feet long and the boards went thru in about 3
minutes (from memory).

This makes me cringe because too much solder paste can be a real
problem and wicking *can* occur.

If that technique works, it's got to be more luck than anything else
imho.
the pic of the beads of paste looked like it was a sloppy way to do
things but the result looked real good. i think the soldermask helped.
I know that working with SMT is a hassle in many ways (especially at my
age and eyesight) but I don't think that trying to make it easy by just
placing solder paste across all the pins of a part is a process that
will be very reliable.

it was the pads. still, i'd just as soon hand solder than do that.
I imagine most of us here just make a few prototypes at a time?

yeah, but my time is valuable and i'd like to have little robotic
foreigners doing my light work for me.
In that case we need something that's as simple and cost effective as
possible, but what's the answer ?

I think that a syringe and a good eyeglass may allow the dispensing of
solder onto each pad of small prototype pcb with reasonable results ?

ouch!! silk screen it.
Or some easy way to produce a silk screen, but I haven't looked into
this yet.

Can some laser printer paper manufacturer please develop a semi stiff
silk A4 sheet please ? ;-)

the best screen turns out to not be silk. it's polyester and that's the
one most commonly used.
Hmm, whats to stop someone using a a4 sticky backed paper sheet and
sticking some silk screen to it, then running thru a laser ?

If the silk melts, its *your problem*, but then again what about a fine

what about getting some screen from a graphic arts supply biz?
If the brass or the cut ends of the brass screen damage the laser toner
roller, once again its *your problem* not mine!

I think there are a few workable ideas here ?

I've got an expendable laser and nearly empty cart, so if I can find the
materials, I'll give it a go as soon as I can
great! if you don't **** it up and it's expendable. send it to me. i
need one.

has been a lot better than that flame war.

talk to ya soon.

brs,
mike

R
Replies
4
Views
2K
Martin Riddle
M
Replies
17
Views
6K
Replies
11
Views
3K
M
Replies
11
Views
3K
josephkk
J
Replies
1
Views
1K


text:
1
AIEEE 2010
+4
-1
The combination of gates shown below yields
A
OR gate
B
NOT gate
C
XOR gate
D
NAND gate
2
AIEEE 2009
+4
-1
The logic circuit shown below has the input waveforms $$'A'$$ and $$'B'$$ as shown. Pick out the correct output waveform.

Output is

A
B
C
D
3
AIEEE 2009
+4
-1
A $$p$$-$$n$$ junction $$(D)$$ shown in the figure can act as a rectifier. An alternating current source $$(V)$$ is connected in the circuit.

The current $$(I)$$ in the resistor $$(R)$$ can be shown by :

A
B
C
D
4
AIEEE 2008
+4
-1
A working transistor with its three legs marked $$P,Q$$ and $$R$$ is tested using a multi-meter. No conduction is found between $$P$$ and $$Q$$. By connecting the common (negative) terminal of the multi-meter to $$R$$ and the other (positive) terminal to $$P$$ or $$Q,$$ some resistance is seen on the multi-meter. Which of the following is true for the transistor?
A
It is an $$npn$$ transistor with $$R$$ as base
B
It is an $$pnp$$ transistor with $$R$$ as collector
C
It is an $$pnp$$ transistor with $$R$$ as emitter
D
It is an $$npn$$ transistor with $$R$$ as collector
JEE Main Subjects
Physics
Mechanics
Electricity
Optics
Modern Physics
Chemistry
Physical Chemistry
Inorganic Chemistry
Organic Chemistry
Mathematics
Algebra
Trigonometry
Coordinate Geometry
Calculus
EXAM MAP
Joint Entrance Examination


text:
# Is the mapping cylinder of a Serre fibration also a Serre fibration?

If we have a Serre fibration $p: E \rightarrow B$ with fiber of homotopy type $S^{k-1}$, then we can create a fibration with contractible fiber by first taking the mapping cylinder $M_p$ of $p$ to get a map $M_p \rightarrow B$ (not necessarily a fibration) with contractible "fiber" and then applying the "space of paths" construction to get a fibration $M_p \times_B B^I \rightarrow B$. My question is, is this last part of the construction necessary, or is the mapping cylinder $M_p$ already a Serre fibration?

I tried lifting a homotopy $f_t: X \times I \rightarrow B$ with starting point $\tilde f_0: X \rightarrow M_p$ by cutting $X$ into the closed preimage $C$ of $B \subset M_p$ and the open preimage $U$ of $E \times [0,1) \subset M_p$. On $C \times I$ we set $\tilde f_t(x) = f_t(x) \in B \subset M_p$. On $U \times I$ we lift $f_t|U: U \times I \rightarrow B$ to $g_t: U \times I \rightarrow E$ and then set $\tilde f_t(x) = (g_t(x),(1-t)(t$ coordinate of $\tilde f_0(x)) + t(1))$. This defines a continuous lift on $C$ and on $U$ separately. If the continuous lift on $U$ extends to the closure of $U$ then we're done. The map $U \rightarrow E$ could be nasty though near the boundary of $U$. Perhaps a better approach is to first construct a map from $X \times I$ that is only "close to" a lift, then use obstruction theory (I'm not an expert on this) to show that it is homotopic to some lift.

-
I suspect that it's not the case. It's a theorem that there exists a functorial factorization into a trivial cofibration and a Serre fibration (which is what you want), but I don't know if one can explicitly compute it, since it results from the small-object argument. –  Harry Gindi Oct 2 '10 at 18:44

Waldhausen, Jahren and myself proved a fiber gluing lemma for Serre fibrations, in the context of simplicial sets, that may be useful. In Propositions 2.7.10 and 2.7.12 of "Spaces of PL manifolds and categories of simple maps"

http://folk.uio.no/rognes/papers/plmf.pdf

we prove that given:

• a diagram of simplicial sets $Z_1 \leftarrowtail Z_0 \to Z_2$, where one map is a cofibration,

• a sufficiently nice base simplicial set $B$ (a simplicial complex will do), and

• compatible maps $Z_i \to B$ that become Serre fibrations upon geometric realization,

then the pushout map $Z_1 \cup_{Z_0} Z_2 \to B$ becomes a Serre fibration upon geometric realization.

Mapping cylinders are a special case of pushouts. If $p \colon E \to B$ becomes a Serre fibration upon realization, then so do the obvious map $E \times \Delta^1 \to B$ and the identity map $B \to B$. The pushout map is your map $M_p \to B$, and our conclusion is that its realization is a Serre fibration.

Our proof depends on working with simplicial sets. The technical condition on $B$ is that each nondegenerate simplex is embedded.

• John
-

I think that your idea works.

You propose "$\tilde f_t(x) = (g_t(x),(1-t)(t$ coordinate of $\tilde f_0(x)) + t(1))$", which I don't quite understand. I would simply do: $\tilde f_t(x) = (g_t(x),t$ coordinate of $\tilde f_0(x))$, where $g_t(x)$ is the solution of the corresponding lifting problem for $p$. Note that you've implicitely used that $U$ is a CW-complex, but there's nothing wrong with that.

As you say, this defines a continuous lift on $C$ and on $U$ separately, and they glue to a continuous lift on the whole.

-
I think I've managed to convince myself that this works by thinking about paths in $U$ that end on $C$. It would be nice if I could finish this proof with more precision though. –  Cary Oct 2 '10 at 21:45
Ah! Here's a potential problem: There are two possible topologies on the mapping cylinder of $E\to B$. The first one is the quotient topology of $E\times [0,1] \sqcup B$. In the second one, you declare any sequence of points whose t-coordinate tends to 1, and whose projection in B converges to converge. I think that my argument shows that the second topology on Mp is a Serre fibration... but I'm a bit confused by now. –  André Henriques Oct 3 '10 at 13:40
You may want to look at Proposition 1.3 in: Mónica Clapp, Duality and transfer for parametrized spectra. Arch. Math. (Basel) 37 (1981), no. 5, 462--472. She proves that a suitable pushout of Hurewicz fibrations over a common base is a Hurewicz fibration. By a theorem of Steinberger and West, a Serre fibration between CW complexes is a Hurewicz fibration, so if $p \colon E \to B$ is a cellular map of CW complexes, and a Serre fibration, then these results prove that $M_p \to B$ is a Hurewicz fibration, hence a Serre fibration. –  John Rognes Oct 4 '10 at 9:09


text:
## Intensity of Sound Waves

The sound level 25 m from a loudspeaker is 71dB. What is the rate at which sound energy is produced by the loudspeaker, assuming it to be an isotropic source? I have no idea where to begin. Please show all of your work.


text:
# [texhax] AtBeginDocument in beamer

Bernice Barnett jbbrus at comcast.net
Fri Apr 2 22:53:47 CEST 2010

Arno Trautmann wrote:
> Hi all!
>
> I’m having some trouble using the beamer class and trying to set frames
> \AtBeginDocument. A minimal example
>
> \documentclass[]{beamer}
>
> \AtBeginDocument{\frame{test1}}
> \begin{document}
> \frame{test2}
> \end{document}
>
> already shows first problems: The navigation symbols are totally wrong
> (at the top of test1, at the bottom of test2).
> Ok, in my document I don’t make use of these symbols, but I have some
> other effects:
>
> • the footline goes anywhere, but not at the bottom. can be corrected by
> using \vspace*
> • the headline leaves about 4mm of blank space above it which cannot be
> corrected by \vspaces
> • \inserttotalframenumber prints just 1 instead of the actual number
>
> The beamer documentation says
>
> • The height of the headline and footline templates is calculated
> automatically. This is done by typesetting the templates and then
> “having a look” at their heights. This recalculation is done right at
> the beginning of the document, after all packages have been loaded and
> even after these have executed their \AtBeginDocument initialization.
>
> So I guess I cannot use \AtBeginDocument to print \frames. Does anybody
> have an advise how to handle this? I need AtBeginDocument as the first
> two frames are generated half-automatically. Is there any internal
> feature I can use (which is stable enough to really use it?) or hack it
> myself anyhow? I could try to use \AtBeginSection instead but then I’d
> have to insert a \section by hand which is useless.
>
> I’d be thankfull for any suggestions,
> cheers
> Arno
>
I'm making a wild guess: Doesn't beamer provide a way to mark a frame as
fragile and, if so, might that be needed here?
--
Jeff Barnett

More information about the texhax mailing list


text:
# Workshop 21. NumPy. Pandas. Data analysis project.¶

## Data Analysis Project¶

2. Check descriptive statistics of the dataset: mean, median, standard deviation for some of the fields (at least 3 fields).
3. Plot graphs for some of the fields (at least 3 fields)
4. Do at least 2 comparisons of values of the fields filtered by some condition (an example of this will be given later).
5. Describe in 2-3 paragraphs, what kind of dataset it is. What data is there, what patterns you noticed in the data.

Full project instructions will be available soon on the wiki.

## Data sources¶

https://www.kaggle.com/

https://dataverse.harvard.edu/

https://github.com/awesomedata/awesome-public-datasets

Other sources. Useful search keywords are "open dataset" or "public dataset".

For the purposes of the project, you should select datasets from areas where "data" is a table of meaningful numbers, instead of something complex like fMRI data, images, eye-tracking data and so on.

## NumPy¶

Numpy is a 3rd party module that provides tools for working with matrices and arrays and performing complex computations.

Its documentation can be found here: https://numpy.org/doc/1.21/

The main object of NumPy is an n-dimensional array.

A 1-dimensional array is a vector. It is similar to a Python list, but it can only hold numbers of the same type.

A 2-dimensional array is a matrix.

In [ ]:
import numpy as np

In [ ]:
# np.array converts a list of lists of numbers into a numpy array
a = np.array([[1.5, 0, 1], [0, 2, 1]])
z = np.zeros((3,4))
print(a)
print(z)

[[1.5 0.  1. ]
[0.  2.  1. ]]
[[0. 0. 0. 0.]
[0. 0. 0. 0.]
[0. 0. 0. 0.]]

In [ ]:
print(a.ndim) # number of dimensions
print(a.shape) # number of elements along each dimension
print(a.size) # number of elements overall
print(a.T) # transposing a matrix

2
(2, 3)
6
[[1.5 0. ]
[0.  2. ]
[1.  1. ]]

In [ ]:
a_single = np.array([1, 2, 9])
a_double = np.array([[1, 2, 9]])
print(a_single)
print(a_double)
print(a_single.T)
print(a_double.T)
print(a_single.shape)
print(a_double.shape)

# making vectors into 1-by-n 2-dimensional arrays simplifies working with them in the context of matrices.

print(a_double.T * a_double)
print(a_single * a_single.T)

[1 2 9]
[[1 2 9]]
[1 2 9]
[[1]
[2]
[9]]
(3,)
(1, 3)
[[ 1  2  9]
[ 2  4 18]
[ 9 18 81]]
[ 1  4 81]

In [ ]:
ab = np.random.rand(3, 4)*10
print(ab)
print()
print()

# Indexing is usually done with tuples
ab_slice = ab[1:,:3]
print(ab_slice)

indices = ab[0, :] > 4
print(indices)
print(ab[:, indices])

[[7.46173146 6.62217718 2.60960204 4.13956425]
[0.73936542 1.88682724 7.96806865 9.35356483]
[4.44482064 2.56019377 6.12784082 8.53571277]]

[[0.73936542 1.88682724 7.96806865]
[4.44482064 2.56019377 6.12784082]]
[ True  True False  True]
[[7.46173146 6.62217718 4.13956425]
[0.73936542 1.88682724 9.35356483]
[4.44482064 2.56019377 8.53571277]]

NumPy ndarrays support basic mathematical operations and perform them much faster than Python lists.

### csv¶

There is a module csv that makes it easier to read and write csv files. However, to work with data itself you have to use standard Python functions.

In [ ]:
### Writing a csv file

import csv

data_list = [
[1, 2, 3],
[4, 5, 6]
]
with open('some.csv', 'w', newline='') as f:
writer = csv.writer(f)
writer.writerows(data_list)

In [ ]:
### Reading a csv file

import csv
data = []
with open('some.csv', newline='') as f:
data.append(row)
print(row)

['1', '2', '3']
['4', '5', '6']

### json¶

Another common data type is json. It stores data similarly to the Python syntax. You can load it with module json.

In [ ]:
### Writing json

import json

data = ['foo', {'bar': ('baz', None, 1.0, 2)}]
with open('some.json', 'w') as f:
json.dump(data, f)

In [ ]:
### Reading json

import json

with open('some.json') as f:
print(data)
print(data[1])

['foo', {'bar': ['baz', None, 1.0, 2]}]
{'bar': ['baz', None, 1.0, 2]}

In the file sales.csv (link) there is a fictional set of sales data for employees of some company.

Load this dataset using the module csv. Convert the numerical parts of data into numpy arrays: training level, work experience, salary, sales.

Compute a numpy array for efficiency that is defined as sales divided by salary. Use only array operations, no loops.

Use the function np.argmax(a) to find which employees have the highest salary, sales value and efficiency.

Use indexing with logical expressions a[a < 5] to find a list of salaries of employees whose work experience is greater than 4 years.

Do not change the order of any of your arrays and use that fact. For each employee, the index in each array is always the same

## Pandas¶

pandas is a module that allows you to perform many tasks related to data analysis.

The main format of storing data in pandas is DataFrame.

### DataFrame¶

DataFrame is a two-dimensional indexed array of values with a header. Generally, rows are objects and columns are individual properties, with their names in the header.

In [ ]:
import pandas as pd

In [ ]:
df_ex_no_labels = pd.DataFrame(ab[1,:])
print(df_ex_no_labels)

          0
0  0.739365
1  1.886827
2  7.968069
3  9.353565

In [ ]:
df_ex = pd.DataFrame(ab[:,:], columns=['Value 1', 'Important number', 'Value 2', 'Another value'])
print(df_ex)

    Value 1  Important number   Value 2  Another value
0  7.461731          6.622177  2.609602       4.139564
1  0.739365          1.886827  7.968069       9.353565
2  4.444821          2.560194  6.127841       8.535713

In [ ]:
# Usually a data frame is read from a file
# comma separated values
data = '''
Example 1,Example 2,Example3
1,2,3
2,51,35
3,0,0
4,50,25

'''

with open('df.csv', 'w') as df_file:
df_file.write(data)

print(df)

   Example 1  Example 2  Example3
0          1          2         3
1          2         51        35
2          3          0         0
3          4         50        25

In [ ]:
df2 = pd.read_csv('https://gist.githubusercontent.com/l8doku/eded81e7f66e504f96117ef90ed28dcf/raw/e286e724378ea8ad47d7ee6ccd245084d7a797cd/example_data.csv')
print(df2)

   Provider Station_ID Composed_Indicator  Overall_Availability  \
0       ANA   51240000               0.35                 19.02
1       ANA   51280000                  -                  0.08
2       BOM    102101A               0.92                 80.09
3       BOM    104001A               0.79                 76.18
4       BOM     304499               0.94                 84.39
5       BOM     402206               0.86                 66.85
6       BOM   A0030501               0.87                 70.94
7      CHDP          1               0.56                 32.05
8      CHDP        100               0.65                 41.54
9      CHDP        101               0.67                 43.48
10     CHDP        102                0.6                 39.41
11     CHDP        103               0.67                 44.07
12     CHDP         57                  -                  7.76
13     CHDP         58               0.67                 38.86
14     CHDP          6               0.72                 44.07

Longest_Availability_without_Gaps  Continuity_in_Available_Data  \
0                                2.17                         11.42
1                                0.07                         88.89
2                               80.09                        100.00
3                               76.18                        100.00
4                               84.39                        100.00
5                               66.85                        100.00
6                               70.94                        100.00
7                                6.49                         20.26
8                               25.28                         60.86
9                               35.02                         80.54
10                              18.28                         46.38
11                              42.37                         96.16
12                               7.76                        100.00
13                               7.05                         18.15
14                              42.37                         96.16

Minimum_Relative_Availability_for_a_Month  Ratio_of_Outliers  \
0                                       94.45               0.93
1                                        0.00               0.00
2                                       99.47               0.74
3                                       98.75               0.67
4                                       98.41               0.55
5                                       98.88               0.76
6                                       97.96               0.69
7                                       96.90               0.19
8                                       97.83               0.26
9                                       97.08               0.83
10                                      94.61               0.62
11                                      99.98               0.75
12                                      87.33               0.12
13                                      97.77               1.03
14                                      99.98               0.25

Homogeneity_of_Yearly_Averages Trend_in_Annual_Flows Trend_in_One_Month
0                               1           Significant        Significant
1                               -                     -                  -
2                               0       Not_significant    Not_significant
3                               0       Not_significant        Significant
4                               0       Not_significant    Not_significant
5                               1       Not_significant    Not_significant
6                               1       Not_significant    Not_significant
7                               0       Not_significant        Significant
8                               0       Not_significant        Significant
9                               0       Not_significant        Significant
10                              0       Not_significant        Significant
11                              1       Not_significant        Significant
12                              -                     -                  -
13                              0       Not_significant    Not_significant
14                              0       Not_significant        Significant

In [ ]:
df2.columns

Out[ ]:
Index(['Provider', 'Station_ID', 'Composed_Indicator', 'Overall_Availability',
'Longest_Availability_without_Gaps', 'Continuity_in_Available_Data',
'Minimum_Relative_Availability_for_a_Month', 'Ratio_of_Outliers',
'Homogeneity_of_Yearly_Averages', 'Trend_in_Annual_Flows',
'Trend_in_One_Month'],
dtype='object')
In [ ]:
df2[['Provider', 'Trend_in_One_Month']]

Out[ ]:
Provider Trend_in_One_Month
0 ANA Significant
1 ANA -
2 BOM Not_significant
3 BOM Significant
4 BOM Not_significant
5 BOM Not_significant
6 BOM Not_significant
7 CHDP Significant
8 CHDP Significant
9 CHDP Significant
10 CHDP Significant
11 CHDP Significant
12 CHDP -
13 CHDP Not_significant
14 CHDP Significant
In [ ]: