Resource Intensive PHP Tasks part 3

The design mock up

 

Simplified design for a long running php process
What we have above is the proposed app overview.

Here is a brief overview of what is actually happening:

  1. Requester (User) Sends a request for a resource to the system. The resource could be HTML, Report etc
  2. Controller receives the requests and returns a HTTP 202 response. See http://en.wikipedia.org/wiki/List_of_HTTP_status_codes for more details on HTTP status codes.
  3. Controller sends job parameters to model
  4. Controllers sends the url that will call the appropriate worker to the queue manager
  5. Queue manager receives job request
  6. Queue manager evaluates systems resources and decides sends requests to workers. This can be done via curl. The queue manager should not wait for a response
  7. Worker does the time intensive task and on completion (successfully or not) sends output back to the Queue manager
  8. The Queue manager updates the progress to the model
  9. Requester(User) sends request for the current progress.
  10. Controller checks progress from the model, if complete returns the resource if not, returns the progress information that is available

Obviously not all work flows will fit this suggested work flow, regardless this should at least come close.

Next time we are going to go through a sample implementation of this design.

If you have any questions please drop a comment below. Remember to follow me on twitter.

by Jacob Chencha

Facebooktwittergoogle_plusredditpinterestlinkedinmail

Resource Intensive PHP Tasks part1

This is the first in  a five part session that will teach you the basics of designing a projects that is resource intensive.

Introduction

So you now have a couple of applications under your belt, maybe even mastered a framework or two. Barring special circumstances you have come to expect response from the server in the order of milliseconds. Then baaaam comes a project with various resource intensive jobs and everything goes whack!!
Below are some of the problems that you are likely to encounter if you don’t plan for this resource hungry tasks:
1. Server will cut you off: The default limit for php scripts is 30 seconds. You could always modify this by changing the execution time http://php.net/manual/en/function.set-time-limit.php and in fact for worker scripts this maybe inevitable but doing this is really just sweeping the problem under the carpet, which is just not a good idea, there are spiders there! Furthermore if you are in a shared scripting environment then things are a tad worse for you because you just don’t have access to php.ini file and are more likely than not unable to change global values.
2. You will run out of resources: Unlike most other scripting languages, php does not repeat file descriptors. For small to medium tasks this is hardly ever a problem for bigger projects however this can escalate very fast. See http://gnuvince.wordpress.com/2008/10/28/php-wrong-for-long-running-processes-wrong-for-america/
3. Google will hate you: While Google’s ranking algorithm remains propriety, it is an open secret that page load does affect rank. Now unless you have a couple of millions to spare in advertising budget, you don’t want to find yourself in the netherworld that is page 2 of Google search!. For interested readers, you may read more here http://www.seochat.com/c/a/google-optimization-help/average-page-load-time-of-top-ranking-websites-in-google/
4. The user will hate you: LOVE YOUR USERS. We can never over emphasize this. Slow and hanging apps amount to right out insulting them. Now you may argue that this is an internal app or that you, the developer, will be the only user but that is irrelevant and besides the point, when developing think of the user as a separate entity. You will thank yourself later.

There is a lot of argument as regards php and task intensive tasks http://stackoverflow.com/questions/2212635/best-way-to-manage-long-running-php-script I think the argument there is irrelevant besides we love PHP through the good and the bad right 🙂

Now that I have presented the challenge let me present the suggested solution. Note what I will be talking about here is design of your application and how to make it ready for long running tasks.

Watch out for part 2 of this series coming to you tomorrow.

If you like this, please follow me on twitter use the tabs to your right.

Thank you

by Jacob Chencha

Facebooktwittergoogle_plusredditpinterestlinkedinmail

Why your first app will fail

So you have just learned your new shiny language (or a couple) and now are wondering what to do next. For most, the options boil down to

  • Join a web development agency (get employed)
  • Become a freelancer
  • Create a new product and sell it.

If you are like me, you probably jumped right into no 3 and started coding your ideas. Hours will be burnt as you toil for many hours at a go until you finally have your product for publishing.

You publish your product on the app markets (if thats what you are into) or buy up a domain for your web app and start drumming up support all the while waiting for users.

Then it happens, Nothing. Not even one single user. What could have gone wrong?

Dunning–Kruger effect

You have just been hit by the Dunning-Kruger effect (D-K)!

The definition according to wikipedia for the D-K effect:

“The Dunning–Kruger effect is a cognitive bias in which unskilled individuals suffer from illusory superiority, mistakenly rating their ability much higher than average. This bias is attributed to a metacognitive inability of the unskilled to recognize their mistakes.”

Now I know you are probably that can’t be right, besides you can’t think of an application that you can’t engineer in you chosen language and you have practically read all the marketing blogs you could get your hands on.

Well unfortunately things are just not that easy and some of the following factors have certainly been playing again you:

  1. You will sub consiously over emphasize on the particular area of expertise that you are good at: For example developers good in scoping and requirements will come up with a very detailed SRS (Sofware requirements document) while those skilled in database design will create great designs but fail to focus on the user experience and so on and so forth.
  2. Feedback will be hard to get: If you have had to maintain someone else’s code or even your own code, you know how much more effort it takes to understand and process it. Couple this with the relatively low number of professional developers, you find yourself in a case where no one gets to see your code and its weaknesses, therego you never improve.
  3. You are smart and you know it: Most developers I have interacted with fall somewhere in the higher end of the bell curve. This probably served them well through their formal years of education. However you (the developer) will very soon realize that while raw intelligence counts for some, market experience counts for so much more, unfortunately it takes a flopped launch (or several) for you to get that.
  4. Low barriers of entry: This probably did serve you well as you we’re starting out. No one cared about your credentials and tutorials we’re a heap. Fortunately this resource is available to all other devs unfortunately most will just never use it. Thus you will have a lot of devs who couldn’t do the fizzbuzz test have already had a run in with you future clients. This is bad because by now they have already developed a negative stereotype to new devs.

Suggested ways to beat it

Here are some of my ideas on how to successfully navigate the muddy waters of D-K.

 

  1. Join a team: If you are a natural extrovert this should be easy however for the rest of us introverts this may take some effort. By joining a team, most of the problems I highlighted will most certainly disappear. If you don’t follow any other advice below, Follow this one.
  2. Give back to the community: Not every project you make needs to be commercial. Open source it! Yes it took time to make but the feedback about the quality of your code and the improvements that can be made will improve your worth as a developer dramatically.
  3. Help new developers: It may seem like a give-give relationship but in reality you may actually be getting more than you are giving. This is because by teaching you are forced to test the assumptions of all that you think you actually do know.
  4. Think small: If you have attended some formal training, you have probably been drummed with hype of huge pay. Even if you are an autodictat you have probably read success stories. The reality however is that what made this entrepreneurs rich was not their first project, something in the range of the 100th or worse. As such don’t pressure yourself too much, you will eventually make it but for now, build light applications and get them coming fast. This will help you build the necessary experience for your big one as well as allow you ample time to make the mistakes that will undoubredly show up.

The more discerning reader may have noticed I have mentioned nothing about spending more time coding your heart away. There is a reason for that. This industry just like all other industries is all about the people.

Your ability to connect with other developers, engineers, designers and most importantly clients is what will truly set you apart.

Thank you.

Other resources

  1. http://computer.howstuffworks.com/5-internet-entrepreneurs.htm
  2. http://www.codinghorror.com/blog/2007/01/how-to-become-a-better-programmer-by-not-programming.html
  3. http://opinionator.blogs.nytimes.com/2010/06/20/the-anosognosics-dilemma-1/?_r=0
  4. http://www.codinghorror.com/blog/2007/02/why-cant-programmers-program.html
  5. http://techeclipse.com/2013/04/10-richest-technology-companies/

If you liked this post and would like to recieve more please click on subscribe and like our page.

Facebooktwittergoogle_plusredditpinterestlinkedinmail

 

Facebooktwittergoogle_plusredditpinterestlinkedinmail