<h3>Thoughts around design systems: implementation — React, Styled-Components, and more</h3>
<h4>Before you jump into building screens, take a step back, and observe all the screens.</h4>
<figure><img alt="A surface pattern design visual just as a preview image for the article." src="https://cdn-images-1.medium.com/max/866/0*-iVHytrm4mUjlHMQ"></figure>For constructing a building, the foundation is a very critical point. On a similar note, investing enough time in understanding the design system and implementing it, will make further development work faster and smoother. A designer provides you with the style guidelines and roughly enough screens to start. This info can guide you in deciding your libraries, packages, and tools you are going to use to style your components. The project complexity and time constraint also could determine your choices. Interaction with the designer to realize his mindset can further greatly influence your way of implementation more logically.
From my experience so far these shall be the fundamental checkpoints when you transform your design systems into code.
<ul>
<li>Readable/Intuitive/logical</li>
<li>Modular</li>
<li>Reusable</li>
<li>Extendable</li>
<li>No over-engineering (Eg Too DRY aka AHA)</li>
</ul>
Well before I start elaborating on the above-mentioned points, let’s for one time ask why even bother a design system? I believe even on just doing a side project where you are the sole developer with great confidence of design sense (puns intended) and no designer, you can make your app shine (in terms of looks here, I meant) just by setting a style guide beforehand and following basic tips and tools. It will naturally make the app look less messy and enable you to give a unique branding to your app.
Basic starter tips
<ul>
<li>Using 4–6 main colors for the app. A random palette generator could be handy <a href="https://coolors.co/">https://coolors.co/</a>
</li>
<li>Focusing on 2–3 font families. <a href="https://fontjoy.com/">FootJoy</a> generates font family sets using AI which could be appealing when used together</li>
<li>Using <a href="https://www.modularscale.com/">https://www.modularscale.com/</a> to generate a set of harmonious font sizes (thanks to Siddharth Kshetrapal for the info)</li>
</ul>
Symmetry is pleasing to the eyes and reduces the cognitive load on the user. In fact, one of the goals behind having a design system is to make your UI consistent which is a very important aspect of UI engineering and branding. This <a href="https://www.interaction-design.org/literature/article/consistency-more-than-what-you-think">article</a> talks extensively about how consistency plays an important role in general for a product.
At the same time, the monotonous design would look boring. Hence some variations enable you to break symmetry and also is important to help the user distinguish between different functionalities or bring attention to specific parts.
Now coming into the main discussion here :)
Below is an example of a relatively simple design system. We will discuss further closely referring to this
<figure><img alt="A screenshot of a simple style guideline for a product. Comprise color palette, border,buttons, form parts, fonts, toast, etc" src="https://cdn-images-1.medium.com/max/1024/1*pSyj-mdSwxgBG4JNFjM4vQ.png"><figcaption>StudioYou style guide</figcaption></figure>StyleGuide specifically provides you with the atoms and molecules you are going to use throughout your application pages. While your various screens guide you in tracing patterns and converting them into building organisms and templates like React components.
In case you are wondering where such chemistry terminology came from. Atomic Design principles say a page can be broken down into templates. Pages → templates → organisms →molecules →atoms
I highly recommend reading about <a href="https://atomicdesign.bradfrost.com/table-of-contents/">atomic design</a> by Brad Fost. In fact, whatever good example of a design system you consider, they are somewhere using ideas of atomic design.
<blockquote>
Atomic Design details all that goes into creating and maintaining robust design systems, allowing you to roll out higher quality, more consistent UIs faster than ever before. This book introduces a methodology for thinking of our UIs as thoughtful hierarchies, discusses the qualities of effective pattern libraries, and showcases techniques to transform your team’s design and development workflow.</blockquote>
Note: my definition of atoms and so may not strictly comply with the atomic design principles as I just draw inspiration from them and do what suits in my case.
As per the above-shared style guide
atoms: Fonts labeled h1,h2,h3,h4, body body-links, paragraphs, etc, the checkboxes, and the icons (close, tick, spinner)
I like the <a href="https://material-ui.com/">Material UI</a> framework as it has highly customizable components and themes. As well you get stuff like accessibility out of the box.
But when you have a full-fledged well-defined design system. Converting the Material UI components into yours is a ton of work. Documentation is great. But still often I had to look into the source code of the components to learn to tweak the styles and match with my desired styles. Moreover using another design system is like promoting their brand not yours exclusively.
Wait, there is something I haven’t yet thought enough. Material Design also has motion design. If your design system also includes a motion or plan to include motion design later, I guess it’s even more hard work to tweak the material design’s motion than to build components from scratch. Google has been building mechanisms for customizing <a href="https://medium.com/google-design/implementing-motion-9f2839002016">motion design</a>. It’s in beta probably.
Owing to the relative simplicity of the scenes and style guide and considering that it wasn’t inspired by Material design, I decided to go with building from the ground up using styled-components. By going through the screens, I realized I may be using Font labeled h2 (style guide) but with tag &lt;h1&gt;. Such scenarios (using the same style but different HTML semantic) are very common especially when you want to be very correct :) and that’s why Styled-Components have got you covered.
Here, we are dealing with (Css-In-Js) way of styling only. If using classes there is no issue in achieving the desired HTML semantic. &lt;h1 class='h2'&gt;Heading&lt;/h1&gt;
But with styled-components, as you see you can the `as` prop to change the semantic.
<pre>export const H2 = styled.h2`</pre>
<pre>font-size: 1.75rem;</pre>
<pre>line-height: 34px;</pre>
<pre>font-weight: bold;</pre>
<pre>color: ${props =&gt;</pre>
<pre>(props.color &amp;&amp; props.theme.color[props.color]) ||</pre>
<pre>props.theme.color.primaryText};</pre>
<pre>`;</pre>
<pre>&lt;Heading&gt;</pre>
<pre>&lt;H2 as='h1'&gt;Appointments&lt;/H2&gt;</pre>
<pre>&lt;/Heading&gt;</pre>
One thing I want to say that you may be tempted to add margins in atomic and molecular components and even more in template components. But you will ultimately end up thinking why the components are not getting positioned in the pages as expected, and rewriting margin attributes anyway and making them not much reusable.
These building blocks should least affect other neighboring components by themselves.
Next coming to
molecules: Buttons, Form Field, Form Label, dropdowns etc. These actually can incorporate the existing atom-like components to form more complex components.
You surely would like to test those before you start plugging them into whole pages. Previously there was no way and we had to directly make and test them on the fly while building pages mostly. But the storybook has enabled us to utilize the power of this component-based model of doing things even more. Hence, I would like to introduce you to Storybook in the context of documenting the atoms and molecules in the developer’s way.
From <a href="https://storybook.js.org/">storybook</a>’s official page
Storybook is a user interface development environment and playground for UI components. The tool enables developers to create components independently and showcase components interactively in an isolated development environment.
Storybook runs outside of the main app so users can develop UI components in isolation without worrying about app-specific dependencies and requirements.
Further, this also gives a developer clearer and quicker exposure of all the existing building blocks aka components at hand. Hence this can be considered as the developer’s way of documenting the given style guide.
You can also document and experiment entire screens design with storybook.
You can use storybook with React or many other popular UI frameworks.
Below is a tiny sample
<figure><img alt="Screenshot of a React Button component documentation done with storybook. Blue button view and description of its props." src="https://cdn-images-1.medium.com/max/1024/1*REZqFWf4Tz-3l2Za_i0beg.png"><figcaption>Storybook Button Component</figcaption></figure>Organisms: Navbar, or Breadcrumbs, Searchbar.
Templates: You can consider them as a reusable layout for a group of molecules, atoms, and organisms).
Some time back in this article I mentioned the terms extendable and No over-engineering. What I’m trying to say is that you can make your components highly configurable more than actually essential or make them so rigid that you encounter a previously missed case and start tweaking your basic components every now and then. A designer might just miss something in the style guide and the developer or designer might find them later and add it.
Hence keeping a balance seems good. There is no tangible/software measuring tool I’m aware of. It basically comes with experience.
A<a href="https://github.com/alexpate/awesome-design-systems">wesome design system</a>s is a list of open-sourced design systems including a lot of popular ones. Going through this is surely gonna give you a very wide perspective.
Further, if you are going to attempt a design system from scratch for the first time, some tiny snippets of how the theme configuration objects could look like can help you decide your own.
1.
<figure><img alt="screen shot of a code snippet containing an app’s theme object" src="https://cdn-images-1.medium.com/max/559/1*tpSYLg7C7BikrdyNXnDFPw.png"></figure>2.
<figure><img alt="screen shot of a code snippet containing an app’s theme object" src="https://cdn-images-1.medium.com/max/428/1*rAZYQXgIztzMt6xDrvRHYA.png"></figure>3. A fairly complex one, too big to be shown here. Below is the link of a theme object of an application build by fully configuring the material UI theme.
<a href="https://github.com/taskcluster/taskcluster/blob/main/ui/src/theme.js">taskcluster/taskcluster</a>
As I dive deeper into UI Engineering, I have been fascinated by the world of design systems. I hope you get at least a bit of insight from this article which you can keep in mind when you are implementing one yourself.
<img src="https://medium.com/_/stat?event=post.clientViewed&amp;referrerSource=full_rss&amp;postId=28ba823682f" width="1" height="1" alt=""><hr>
<a href="https://uxdesign.cc/thoughts-around-design-systems-implementation-react-styled-components-etc-and-more-28ba823682f">Thoughts around design systems: implementation — React, Styled-Components, and more</a> was originally published in <a href="https://uxdesign.cc/">UX Collective</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.

Thoughts around design systems: implementation — React, Styled-Components, and more

Well GraphQL Asia 2020 is happening in February, hence I thought how about sharing about my experience of the last GraphQL Asia this year.
<figure><img alt="" src="https://cdn-images-1.medium.com/max/225/1*zDPJd09UOExlYuDOLRzCcg.png"><figcaption>GraphQL</figcaption></figure>When I first heard of GraphQL Asia, I felt excited that an international conference is going to take place in Bengaluru, India with so many speakers (info on that in some time). But then I hesitated to apply for the student/diversity scholarship since I haven’t tried GraphQL by that time. I had a vague conception/misconception that it is some server-side or database thing used by facebook and many other applications to tackle very high traffic. But very soon I had to learn about apollo client, GraphQL etc owing to incorporating some new features in the Taskcluster UI and its server (Open Source) :).
<figure><img alt="" src="https://cdn-images-1.medium.com/max/881/1*IuScfOJMWpqarqNJilFvzw.png"></figure>Now I can say what really is GraphQL and had started loving it. Luckily scholarship applications were open , so just went for it.
<figure><img alt="" src="https://cdn-images-1.medium.com/max/292/1*jI6xOwZ_8sqFizjFgj4zDw.jpeg"></figure>If the<a href="https://graphql.asia/2019/"> link i</a>s still live, check out the speaker’s lineup 🔥. I have to say the speakers were diverse ranging from companies like Atlassian, Microsoft, Twitter, Amazon, Myntra, Paypal to startups as small as one just being the only developer. 30+ talks and some workshops in just 2 days and swags of course. Too much to consume.
Tip: During talks jot down things somewhere what you find interesting, any term, tool or concept which you would like to deep dive further.
If you have no clue of what GraphQL is and I’m to introduce you in few words here’s my take to make at least an iota of sense
<blockquote>GraphQL is a query language which allows the client to ask for exactly the data required and that too in one shot instead of making several rest api calls thus saving several client to server roundtrips.</blockquote>
There were talks of adopting GraphQL in their own ecosystems, various GraphQL tools and tacking various challenges.
Now I would like to mention a few talks. Each talk was awesome and enjoyed. So I actually can’t compare. But I ‘m just taking these to give you an overview.
The Case Against GraphQL — Robert Zhu
This was the second talk after the opening talk of the conf by <a href="https://medium.com/u/d9b1a61823fa">Lee Byron</a> (co-creator of GraphQL) who recited a story of how the web has evolved over the past two decades.
Key Points
Interestingly <a href="https://medium.com/u/3e6b69951eef">Robert Zhu</a> spoke about the many arguments
The Novelty/Cargo cult which shouldn’t be followed — Amazon uses microservices and it is successful. Facebook uses React and it is successful. Hence React and Microservices means success!
He spoke about many arguments and challenges which can be faced while dealing with GraphQL
GraphQL has limited data types. It has 32 bit Int as the only integer type. Whereas most languages have 64bit long.
The impedance mismatch case (eg graphQL supporting nullability in languages lacking null support directly)
Then how high bandwidth due to deeply nested queries can be solved with persistent queries
GraphQL execution in-depth — Andreas Marek
This talk would be helping for beginners to understand how the graphQL works as he goes on explaining from the basics of a resolver, coercion, merger field, fragments and how the graphQL engine tackle if a non nullable field receives a null value, by setting the parent null if its nullable or else bubbles up further unless finds a nullable parent (something I didn’t know)
<figure><img alt="" src="https://cdn-images-1.medium.com/max/861/1*AZm-aBBuOMfAj3prc_XjZg.png"></figure>And finally integrating all the concepts into showing the execution algorithm flow for custom scalars, objects and unions, lists. Finally explaining the resolution order
You can check out the graphQL <a href="https://github.com/graphql/graphql-spec/blob/master/spec/Section%206%20--%20Execution.md">specs</a>.
200 OK! Error Handling in GraphQL — Sasha Solomon
Honestly, I was most eagerly waiting for this one. I got stuck abit on how to best tackle errors in GraphQL and after this talk soon I put what I learnt into practice <a href="https://github.com/taskcluster/taskcluster/pull/466">https://github.com/taskcluster/taskcluster/pull/466</a>. Although it added complexity in my case and didn’t fit very well, so dropped that suggestion to simpler means.
From graphql server errors are returned in an array and unless a network error everything else returned from graphQL is kinda status code 200. Take an example of how error data looks like below
<pre>"errors": [ { "message": "Name for character with ID 1002 could not be fetched.", "locations": [ { "line": 6, "column": 7 } ], "path": [ "hero", "heroFriends", 1, "name" ] } ]</pre>
<figure><img alt="" src="https://cdn-images-1.medium.com/max/553/1*oKnzcKIPvsJqfj5eoy4IGg.png"><figcaption>Slide from talk</figcaption></figure>Now the unexpected errors generally with the status codes in 500 range returned from different microservices or apis are something for which the actual user of the app shouldn’t bother exactly what is the error. Hence the client can show a general message. Now for the errors which the app user are more concerned and errors which are more defined can be managed in a more strategic manner. Taking an example of the code below, The resultant return data has a union type of types User( data asked for), and other different defined scenarios - Is Blocked, Suspended, Deleted.
<pre>union UserResult = User| Deleted |Is Blocked | Suspended</pre>
Cleartrip reactive journey from 3 to 0.2s using Apollo GraphQL — a Case Study — Eesh Tyagi
This talk is about adopting Apollo GraphQL in ClearTrip, their new architecture and the drastic performance improvements
<ul>
<li>Page Load time decreased from 9s to 3s</li>
<li>Render Time 3s to 0.2s</li>
<li>Page Size decreased from 1.5Mb to 245kb</li>
</ul>
GraphQL Test-Automation with Karate — Peter Thomas
This was the only talk about testing. There is an increase in the number of StackOverflow questions on Karate. Seems like more people are trying it out. I have a bit of testing experience and I found some use cases quite relevant to me. Few key takeaways which I will look into when testing with any framework would be Testing specific sections of the JSON data using JSON Paths and regex.
Do performance testing with less redundant code (In fact I have wondered on it a lot but never deep dived into this)
GraphQL design patterns for real-time and offline app architecture — Richard Threlkeld
The last talk but one of the most content-heavy talks is must watch as Ricard explains the architecture and concepts to keep offline apps in sync. Lambda architecture, thoughts on how caches and stores are different. This talk is all about the data inconsistency problems and performance optimisations in real offline apps having around 1000 users and what strategies they adopted to deal with it.
These were some other talks which were completely new to me.
Building a high-performance real-time GraphQL API on existing databases — Tanmai Gopal
Serverless GraphQL — Shawn Wang
Live Queries implementation with Apollo &amp; GraphQL — Kiran Kumar Abburi
And yeah met one of the Hasura’s co-founder Tanmai. I took an online MOOC called IMAD taught by him on app development in 2016 which was also my first step towards web development. So finally, met in person. Kudos to his teaching skills
<figure><img alt="" src="https://cdn-images-1.medium.com/max/326/1*ocwqJaRiORbKdtpJsHon7g.jpeg"><figcaption>Left- Me, Right- Tanmai</figcaption></figure>What I found from this conference is that there is massive craze around GraphQL and soon more and more developers would be adopting it. It is just the beginning. With an ever increasing internet users and that too on mobile devices. GraphQL is there to the some rescue.
<h3>Some slide and workshop links</h3>
<ul>
<li><a href="https://speakerdeck.com/tgriesser/paving-a-new-path-for-graphql-schemas">https://speakerdeck.com/tgriesser/paving-a-new-path-for-graphql-schemas</a></li>
<li>
<a href="https://drive.google.com/file/d/1ek8aqhAiJyBslzJu4nNN4uYxxKdkwjGq/view?">https://drive.google.com/file/d/1ek8aqhAiJyBslzJu4nNN4uYxxKdkwjGq/view?</a> usp=sharing <a href="http://bit.ly/glaphqlasia2019ken">http://bit.ly/glaphqlasia2019ken</a>
</li>
<li><a href="https://github.com/gsans/aws-amplify-workshop-angular">https://github.com/gsans/aws-amplify-workshop-angular</a></li>
<li>
<a href="https://drive.google.com/file/d/1nQ5DKdEjomkOV_oA48Q0UwGMb8g_Paf_/">https://drive.google.com/file/d/1nQ5DKdEjomkOV_oA48Q0UwGMb8g_Paf_/</a> view?usp=drivesdk</li>
<li>
<a href="https://docs.google.com/presentation/d/">https://docs.google.com/presentation/d/</a> 128xcLPP2gdRuA3nFYKG9dpyxykzD0sn94c14L3KsX04/edit?usp=sharing</li>
<li>
<a href="https://docs.google.com/presentation/d/">https://docs.google.com/presentation/d/</a> 1Pl0Yjs6hj2jqScW5f0WjErXtZjGzrNLXDKKAwLkEFKU/edit?usp=drivesdk</li>
<li>
<a href="https://www.slideshare.net/TobiasMeixner/brikl-a-graphql-native-graphql-">https://www.slideshare.net/TobiasMeixner/brikl-a-graphql-native-graphql-</a> asia-2019</li>
<li>
<a href="https://drive.google.com/file/d/1ek8aqhAiJyBslzJu4nNN4uYxxKdkwjGq/view?">https://drive.google.com/file/d/1ek8aqhAiJyBslzJu4nNN4uYxxKdkwjGq/view?</a> usp=sharing</li>
</ul>
<h3>GraphQL Asia Channel</h3>
<a href="https://www.youtube.com/channel/UCXMN61PUm8_U-GYMIxffQkA">GraphQL Asia</a>
Comment what I can add to share conference experiences better the next time.
You can find me on twitter <a href="https://twitter.com/Biboswan98">@Biboswan98</a>. DM’s are open.
<img src="https://medium.com/_/stat?event=post.clientViewed&amp;referrerSource=full_rss&amp;postId=637a83835dcf" width="1" height="1" alt="">

GraphQL Asia 2019

First let’s talk what do you mean by optimising a model. Well simply we want the model to get trained to reach the state of maximum accuracy given resource constraints like time, computing power, memory etc. Optimization has a broad scope and you can also tweak the architecture to get a better model. But that is something which comes with intuition developed by experience.
<figure><img alt="" src="https://cdn-images-1.medium.com/max/620/1*Fo7HjwOjJpWoZzw-UPmioA.png"></figure>Given a certain architecture, in pytorch a torch.optim package implements various optimization algorithms. We would discuss here two most widely used optimizing techniques stochastic gradient descent (optim.SGD) and Adam’s Method (optim.Adam).
<ol><li>SGD:</li></ol>
We know that gradient descent is the rate of loss function w.r.t the weights a.k.a model parameters. The loss function can be a function of the mean square of the losses accumulated over the entire training dataset. Hence the weights are updated once at the end of each epoch. This results in reaching the exact minimum but requires heavy computation time/epochs to reach that point.
On the other hand in SGD the weights are updated after looping via each training sample.
<pre>for i in range(nb_epochs): np.random.shuffle(data) for example in data: params_grad = evaluate_gradient(loss_function, example, params) params = params - learning_rate * params_grad</pre>
From official documentation of pytorch SGD function has the following definition
torch.optim.SGD(params, lr=&lt;required parameter&gt;, momentum=0, dampening=0, weight_decay=0, nesterov=False)
Arguments:
<ul>
<li>
params (iterable) — iterable of parameters to optimize or dicts defining parameter groups</li>
<li>
lr (<a href="https://docs.python.org/3/library/functions.html#float">float</a>) — learning rate</li>
<li>
momentum (<a href="https://docs.python.org/3/library/functions.html#float">float</a>, optional) — momentum factor (default: 0)</li>
<li>
weight_decay (<a href="https://docs.python.org/3/library/functions.html#float">float</a>, optional) — weight decay (L2 penalty) (default: 0)</li>
<li>
dampening (<a href="https://docs.python.org/3/library/functions.html#float">float</a>, optional) — dampening for momentum (default: 0)</li>
<li>
nesterov (<a href="https://docs.python.org/3/library/functions.html#bool">bool</a>, optional) — enables Nesterov momentum (default: False)</li>
</ul>
Most of the arguments stated above I believe are self explanatory except momemtum and nesterov.
<figure><img alt="" src="https://cdn-images-1.medium.com/proxy/1*f9a162GhpMbiTVTAua_lLQ.png"></figure>Normally we would see that while training the model, loss decreases immediately in the starting but gradually you reach a point when it seems you aren’t making any progress at all!. So what is wrong?
The problem could be the optimizer’s old nemesis, pathological curvature. Pathological curvature is, simply put, regions of f which aren’t scaled properly. The landscapes are often described as valleys, trenches, canals and ravines. The iterates either jump between valleys, or approach the optimum in small, timid steps. Progress along certain directions grind to a halt. In these unfortunate regions, gradient descent fumbles.
Here using momemtum comes to the rescue. Correct value of momentum is obtained by cross validation and would avoid getting stuck in a local minima.
<pre>v = mu * v - learning_rate * dx # integrate velocity x += v # integrate position </pre>
<figure><img alt="" src="https://cdn-images-1.medium.com/max/562/0*_cXuLABFac4S1aPX"><figcaption>Stuck in a local minima</figcaption></figure>Nesterov Momentum
This type of momemtum has a slightly different methodology. Here weights update depend both on the classical momemtun and the gradient step in future with the present momemtum. The diagram below clearly depicts I’m trying to say.
<figure><img alt="" src="https://cdn-images-1.medium.com/max/800/0*F9jrNcSEy-N-3PmY.jpeg"></figure><pre>x_ahead = x + mu * v # evaluate dx_ahead (the gradient at x_ahead instead of at x) v = mu * v - learning_rate * dx_ahead x += v</pre>
In fact it is said that SGD+Nesterov can be as good as Adam’s technique.
You can totally skip the details because in code you only need to pass values to the arguments. But it is good to know in dept of everything we want to learn.
2. Adam (Adaptive Momemt Estimation)
From the official documentation
<pre>torch.optim.Adam(params, lr=0.001, betas=(0.9, 0.999), eps=1e-08, weight_decay=0, amsgrad=False)</pre>
Arguments:
<ul>
<li>
params (iterable) — iterable of parameters to optimize or dicts defining parameter groups</li>
<li>
lr (<a href="https://docs.python.org/3/library/functions.html#float">float</a>, optional) — learning rate (default: 1e-3)</li>
<li>
betas (Tuple[<a href="https://docs.python.org/3/library/functions.html#float">float</a>, <a href="https://docs.python.org/3/library/functions.html#float">float</a>], optional) — coefficients used for computing running averages of gradient and its square (default: (0.9, 0.999))</li>
<li>
eps (<a href="https://docs.python.org/3/library/functions.html#float">float</a>, optional) — term added to the denominator to improve numerical stability (default: 1e-8)</li>
<li>
weight_decay (<a href="https://docs.python.org/3/library/functions.html#float">float</a>, optional) — weight decay (L2 penalty) (default: 0)</li>
<li>
amsgrad (boolean, optional) — whether to use the AMSGrad variant of this algorithm from the paper <a href="https://openreview.net/forum?id=ryQu7f-RZ">On the Convergence of Adam and Beyond</a> (default: False)</li>
</ul>
Adam’s method considered as a method of Stochastic Optimization is a technique implementing adaptive learning rate. Whereas in normal SGD the learning rate has an equivalent type of effect for all the weights/parameters of the model.
Hm, let me show you the actual equations for Adam’s to give you an intuition of the adaptive learning rate per paramter.
<pre>m = beta1*m + (1-beta1)*dx mt = m / (1-beta1**t) v = beta2*v + (1-beta2)*(dx**2) vt = v / (1-beta2**t) x += - learning_rate * mt / (np.sqrt(vt) + eps)</pre>
We see that Adam somewhat implies two tricks one is momemtum
Another trick that Adam uses is to adaptively select a separate learning rate for each parameter. Parameters that would ordinarily receive smaller or less frequent updates receive larger updates with Adam (the reverse is also true). This speeds learning in cases where the appropriate learning rates vary across parameters. For example, in deep networks, gradients can become small at early layers, and it make sense to increase learning rates for the corresponding parameters. Another benefit to this approach is that, because learning rates are adjusted automatically, manual tuning becomes less important. Standard SGD requires careful tuning (and possibly online adjustment) of learning rates, but this less true with Adam and related methods. It’s still necessary to select hyperparameters, but performance is less sensitive to them than to SGD learning rates. — Stackoverflow
I ‘m writing this article myself as I’m diving deeper into deep learning fundamentals and concepts. Any suggestions on making the article better will be highly appreciated. I hope this article was of some help to you.
<img src="https://medium.com/_/stat?event=post.clientViewed&amp;referrerSource=full_rss&amp;postId=8dbf1890fbdc" width="1" height="1" alt="">

Blog

Below are some of my articles from other platforms. Any new article short or long on tech or other thought shall appear here in my blog now.

Thoughts around design systems: implementation — React, Styled-Components, and more

Read Time: 7mins

GraphQL Asia 2019

Read Time: 6mins

optim.Adam vs optim.SGD. Let’s dive in

Read Time: 5mins